Image processing apparatus and method, and image-capturing apparatus based on the difference between a signal detected by a sensor and the real world

ABSTRACT

An apparatus for detecting a mixed area in frames. An object extracting unit extracts a foreground object from an input image and generates an area-specified object formed of the foreground object and a value indicating that the foreground object belongs to a background area. A motion compensator compensates for the motion of the area-specified object based on a motion vector and positional information thereof. A subtracting unit subtracts the pixel value of a pixel belonging to a foreground object of a current frame from the corresponding pixel of the foreground object of a preceding frame, so as to obtain a frame difference between the pixels belonging to the foreground area. A threshold-value processor detects a mixed area based on the difference.

TECHNICAL FIELD

The present invention relates to image processing apparatuses andmethods, and image-capturing apparatuses, and more particularly, to animage processing apparatus and method, and an image-capturing apparatusin which a difference between a signal detected by a sensor and the realworld is taken into consideration.

BACKGROUND ART

A technique for detecting incidents occurring in the real world by asensor and for processing sampled data output from the image sensor iswidely used.

For example, motion blur occurs in an image obtained by capturing anobject moving in front of a predetermined stationary background with avideo camera if the moving speed is relatively high.

However, when an object is moving in front of a stationary background,not only does motion blur caused by the mixture of the moving objectitself occur, but also the mixture of the background image and theobject image occurs. Hitherto, a process for handling the mixture stateof the background image and the moving object has not been considered.

DISCLOSURE OF INVENTION

The present invention has been made in view of the above-describedbackground. Accordingly, it is an object of the present invention tomake it possible to detect an area in which the mixture occurs.

An image processing apparatus of the present invention includes motioncompensation means for compensating for the motion of frames of imagedata; and area detection means for detecting a mixed area based on thedifference between the pixel data at the corresponding position in themotion-compensated frames.

The area detection means may detect the mixed area to which at leastpixel data belongs when the difference is greater than or equal to athreshold.

The area detection means may further detect, based on temporal change ofthe detected mixed area, a covered background area in which a foregroundobject component corresponding to a foreground increases over time andan uncovered background area in which a background object componentcorresponding to a background increases over time.

The area detection means may further detect, based on a motion vectorcorresponding to the pixel data in each of the frames, a coveredbackground area in which a foreground object component corresponding toa foreground increases over time and an uncovered background area inwhich a background object component corresponding to a backgroundincreases over time.

The image processing apparatus may further include motion vectordetection means for detecting the motion vector.

The image processing apparatus may further include mixture-ratiocalculation means for calculating a mixture ratio indicating the statein which the objects are mixed in the pixel data.

The image processing apparatus may further include separation means forseparating at least a foreground object component corresponding to aforeground from the pixel data of the mixed area based on the mixtureratio.

The image processing apparatus may further include motion-blur adjustingmeans for adjusting the amount of motion blur in the separatedforeground object component.

The image processing apparatus may further include synthesizing meansfor synthesizing another desired object with the separated foregroundobject component based on the mixture ratio.

The motion compensation means may perform motion compensation byshifting a peripheral frame around a designated frame so that abackground object in the designated frame is disposed at the same pixelposition as the background object in the peripheral frame. The areadetection means may detect at least the mixed area based on thedifference between the motion-compensated peripheral frame and thedesignated frame.

The area detection means may be provided with stationary/movingdetermination means for performing a stationary or moving determinationbased on the difference between the pixel data at the correspondingpixel position in the motion-compensated peripheral frame and thedesignated frame. Based on the determination of the stationary/movingdetermination means, the area detection means may detect in which of aforeground area formed of only a foreground object component forming theforeground object in the plurality of objects, a background area formedof only a background object component forming the background object, orthe mixed area the pixel position is.

The area detection means may specify an uncovered background area and acovered background area in the mixed area based on the determination ofthe stationary/moving determination means, the uncovered background areabeing formed at the trailing end in the direction in which theforeground object is moving in the mixed area, the covered backgroundarea being formed at the leading end in the direction in which theforeground object is moving.

An image processing method of the present invention includes a motioncompensating step of compensating for the motion of frames of imagedata; and an area detecting step of detecting a mixed area based on thedifference between the pixel data at the corresponding position in themotion-compensated frames.

In the area detecting step, the mixed area to which at least pixel databelongs may be detected when the difference is greater than or equal toa threshold.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on temporal change of the detected mixed area.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on a motion vector corresponding to the pixel data ineach of the frames.

The image processing method may further include a motion vectordetecting step of detecting the motion vector.

The image processing method may further include a mixture-ratiocalculating step of calculating a mixture ratio indicating the state inwhich the objects are mixed in the pixel data.

The image processing method may further include a separating step ofseparating at least a foreground object component corresponding to aforeground from the pixel data of the mixed area based on the mixtureratio.

The image processing method may further include a motion-blur adjustingstep of adjusting the amount of motion blur in the separated foregroundobject component.

The image processing method may further include a synthesizing step ofsynthesizing another desired object with the separated foreground objectcomponent based on the mixture ratio.

In the motion compensating step, motion compensation may be performed byshifting a peripheral frame around a designated frame so that abackground object in the designated frame is disposed at the same pixelposition as the background object in the peripheral frame. In the areadetecting step, at least the mixed area may be detected based on thedifference between the motion-compensated peripheral frame and thedesignated frame.

The area detecting step may include a stationary/moving determining stepof performing a stationary or moving determination based on thedifference between the pixel data at the corresponding pixel position inthe motion-compensated peripheral frame and the designated frame. In thearea detecting step, it may be detected based on the determination inthe stationary/moving determining step in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.

In the area detecting step, an uncovered background area and a coveredbackground area in the mixed area may be specified based on thedetermination in the stationary/moving determining step, the uncoveredbackground area being formed at the trailing end in the direction inwhich the foreground object is moving, the covered background area beingformed at the leading end in the direction in which the foregroundobject is moving.

A program in a recording medium of the present invention includes amotion compensating step of compensating for the motion of frames ofimage data; and an area detecting step of detecting a mixed area basedon the difference between the pixel data at the corresponding positionin the motion-compensated frames.

In the area detecting step, the mixed area to which at least pixel databelongs may be detected when the difference is greater than or equal toa threshold.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on temporal change of the detected mixed area.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on a motion vector corresponding to the pixel data ineach of the frames.

The program may further include a motion vector detecting step ofdetecting the motion vector.

The program may further include a mixture-ratio calculating step ofcalculating a mixture ratio indicating the state in which the objectsare mixed in the pixel data.

The program may further include a separating step of separating at leasta foreground object component corresponding to a foreground from thepixel data of the mixed area based on the mixture ratio.

The program may further include a motion-blur adjusting step ofadjusting the amount of motion blur in the separated foreground objectcomponent.

The program may further include a synthesizing step of synthesizinganother desired object with the separated foreground object componentbased on the mixture ratio.

In the motion compensating step, motion compensation may be performed byshifting a peripheral frame around a designated frame so that abackground object in the designated frame is disposed at the same pixelposition as the background object in the peripheral frame. In the areadetecting step, at least the mixed area may be detected based on thedifference between the motion-compensated peripheral frame and thedesignated frame.

The area detecting step may include a stationary/moving determining stepof performing a stationary or moving determination based on thedifference between the pixel data at the corresponding pixel position inthe motion-compensated peripheral frame and the designated frame. In thearea detecting step, it may be detected based on the determination inthe stationary/moving determining step in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.

In the area detecting step, an uncovered background area and a coveredbackground area in the mixed area may be specified based on thedetermination in the stationary/moving determining step, the uncoveredbackground area being formed at the trailing end in the direction inwhich the foreground object is moving, the covered background area beingformed at the leading end in the direction in which the foregroundobject is moving.

A program of the present invention causes a computer to execute a motioncompensating step of compensating for the motion of frames of imagedata; and an area detecting step of detecting a mixed area based on thedifference between the pixel data at the corresponding position in themotion-compensated frames.

In the area detecting step, the mixed area to which at least pixel databelongs may be detected when the difference is greater than or equal toa threshold.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on temporal change of the detected mixed area.

In the area detecting step, a covered background area in which aforeground object component corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent corresponding to a background increases over time may furtherbe detected based on a motion vector corresponding to the pixel data ineach of the frames.

The program may further include a motion vector detecting step ofdetecting the motion vector.

The program may further include a mixture-ratio calculating step ofcalculating a mixture ratio indicating the state in which the objectsare mixed in the pixel data.

The program may further include a separating step of separating at leasta foreground object component corresponding to a foreground from thepixel data of the mixed area based on the mixture ratio.

The program may further include a motion-blur adjusting step ofadjusting the amount of motion blur in the separated foreground objectcomponent.

The program may further include a synthesizing step of synthesizinganother desired object with the separated foreground object componentbased on the mixture ratio.

In the motion compensating step, motion compensation may be performed byshifting a peripheral frame around a designated frame so that abackground object in the designated frame is disposed at the same pixelposition as the background object in the peripheral frame. In the areadetecting step, at least the mixed area may be detected based on thedifference between the motion-compensated peripheral frame and thedesignated frame.

The area detecting step may include a stationary/moving determining stepof performing a stationary or moving determination based on thedifference between the pixel data at the corresponding pixel position inthe motion-compensated peripheral frame and the designated frame. In thearea detecting step, it may be detected based on the determination inthe stationary/moving determining step in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.

In the area detecting step, an uncovered background area and a coveredbackground area in the mixed area may be specified based on thedetermination in the stationary/moving determining step, the uncoveredbackground area being formed at the trailing end in the direction inwhich the foreground object is moving, the covered background area beingformed at the leading end in the direction in which the foregroundobject is moving.

An image-capturing apparatus of the present invention includesimage-capturing means for outputting a subject image captured by animage-capturing device including a predetermined number of pixels havinga time integrating function as image data consisting of a predeterminednumber of pixel data; motion compensation means for compensating for themotion of frames of image data; and area detection means for detecting,based on the difference between the pixel data at the correspondingposition in the motion-compensated frames, a mixed area from the imagedata in which a plurality of objects are mixed in the real world andwhich is obtained as the image data.

The area detection means may detect the mixed area to which at leastpixel data belongs when the difference is greater than or equal to athreshold.

The area detection means may further detect, based on temporal change ofthe detected mixed area, a covered background area in which a foregroundobject component corresponding to a foreground increases over time andan uncovered background area in which a background object componentcorresponding to a background increases over time.

The area detection means may further detect, based on a motion vectorcorresponding to the pixel data in each of the frames, a coveredbackground area in which a foreground object component corresponding toa foreground increases over time and an uncovered background area inwhich a background object component corresponding to a backgroundincreases over time.

The image-capturing apparatus may further include motion vectordetection means for detecting the motion vector.

The image-capturing apparatus may further include mixture-ratiocalculation means for calculating a mixture ratio indicating the statein which the objects are mixed in the pixel data.

The image-capturing apparatus may further include separation means forseparating at least a foreground object component corresponding to aforeground from the pixel data in the mixed area based on the mixtureratio.

The image-capturing apparatus may further include motion-blur adjustingmeans for adjusting the amount of motion blur in the separatedforeground object component.

The image-capturing apparatus may further include synthesizing means forsynthesizing another desired object with the separated foreground objectcomponent based on the mixture ratio.

The motion compensation means may perform motion compensation byshifting a peripheral frame around a designated frame so that abackground object in the designated frame is disposed at the same pixelposition as the background object in the peripheral frame. The areadetection means may detect at least the mixed area based on thedifference between the motion-compensated peripheral frame and thedesignated frame.

The area detection means may be provided with stationary/movingdetermination means for performing a stationary or moving determinationbased on the difference between the pixel data at the correspondingpixel position in the motion-compensated peripheral frame and thedesignated frame. Based on the determination of the stationary/movingdetermination means, the area detection means may detect in which of aforeground area formed of only a foreground object component forming theforeground object in the plurality of objects, a background area formedof only a background object component forming the background object, orthe mixed area the pixel position is.

The area detection means may specify an uncovered background area and acovered background area in the mixed area based on the determination ofthe stationary/moving determination means, the uncovered background areabeing formed at the trailing end in the direction in which theforeground object is moving in the mixed area, the covered backgroundarea being formed at the leading end in the direction in which theforeground object is moving.

The motion of frames of image data is compensated for, and a mixed areais detected based on the difference between pixel data at thecorresponding position in the motion-compensate frames.

This enables a mixed area in which mixture occurs to be detected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an embodiment of a signal processing apparatusaccording to the present invention.

FIG. 2 is a block diagram illustrating the signal processing apparatus.

FIG. 3 illustrates the image capturing performed by a sensor.

FIG. 4 illustrates the arrangement of pixels.

FIG. 5 illustrates the operation of a detection device.

FIG. 6A illustrates an image obtained by image-capturing an objectcorresponding to a moving foreground and an object corresponding to astationary background.

FIG. 6B illustrates a model of an image obtained by image-capturing anobject corresponding to a moving foreground and an object correspondingto a stationary background.

FIG. 7 illustrates a background area, a foreground area, a mixed area, acovered background area, and an uncovered background area.

FIG. 8 illustrates a model obtained by expanding in the time directionthe pixel values of pixels aligned side-by-side in an image obtained byimage-capturing an object corresponding to a stationary foreground andan the object corresponding to a stationary background.

FIG. 9 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 10 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 11 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 12 illustrates an example in which pixels in a foreground area, abackground area, and a mixed area are extracted.

FIG. 13 illustrates the relationships between pixels and a modelobtained by expanding the pixel values in the time direction.

FIG. 14 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 15 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 16 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 17 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 18 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 19 is a flowchart illustrating the processing for adjusting theamount of motion blur.

FIG. 20 is a block diagram illustrating the configuration of an areaspecifying unit 103.

FIG. 21 is a block diagram illustrating the configuration of the areaspecifying unit 103 in more detail.

FIG. 22 illustrates a process made by a motion capturing portion 222.

FIG. 23 illustrates an image when an object corresponding to aforeground is moving.

FIG. 24 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 25 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 26 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 27 illustrates the conditions for determining the area.

FIG. 28A illustrates an example of the result obtained by specifying thearea by the area specifying unit 103.

FIG. 28B illustrates an example of the result obtained by specifying thearea by the area specifying unit 103.

FIG. 28C illustrates an example of the result obtained by specifying thearea by the area specifying unit 103.

FIG. 28D illustrates an example of the result obtained by specifying thearea by the area specifying unit 103.

FIG. 29 illustrates an example of the result obtained by specifying thearea by the area specifying unit 103.

FIG. 30 is a flowchart illustrating the area specifying processing.

FIG. 31 is a flowchart illustrating the area specifying processing.

FIG. 32 is a block diagram illustrating another configuration of thearea specifying unit 103.

FIG. 33 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 34 illustrates an example of an area specified object.

FIG. 35 illustrates an example of a motion-compensated area specifiedobject.

FIG. 36 illustrates an example of the processing made by athreshold-value processor 255.

FIG. 37 is a block diagram illustrating the configuration of a timechange detector 256.

FIG. 38 illustrates the determination processing made by the time changedetector 256.

FIG. 39 illustrates the determination processing made by the time changedetector 256.

FIG. 40 illustrates the conditions for determining the mixed area.

FIG. 41 is a flowchart illustrating the area specifying processingperformed by the area specifying unit 103.

FIG. 42 is a flowchart illustrating in detail the processing fordetecting a covered background area or an uncovered background area.

FIG. 43 is a block diagram illustrating still another configuration ofthe area specifying unit 103.

FIG. 44 illustrates the determination processing performed by anidentification unit 281.

FIG. 45 is a flowchart illustrating in detail the processing fordetecting a covered background area or an uncovered background area.

FIG. 46 is a block diagram illustrating an example of the configurationof a mixture-ratio calculator 104.

FIG. 47 illustrates an example of the ideal mixture ratio α.

FIG. 48 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 49 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 50 illustrates the approximation using the correlation betweenforeground components.

FIG. 51 illustrates the relation between C, N, and P.

FIG. 52 is a block diagram illustrating the configuration of anestimated-mixture-ratio processor 401.

FIG. 53 illustrates an exemplary estimated mixture ratio.

FIG. 54 is a block diagram illustrating another configuration of themixture-ratio calculator 104.

FIG. 55 is a flowchart illustrating the mixture-ratio calculationprocessing.

FIG. 56 is a flowchart illustrating the processing for calculating theestimated mixture ratio.

FIG. 57 illustrates a straight line for approximating the mixture ratioα.

FIG. 58 illustrates a plane for approximating the mixture ratio α.

FIG. 59 illustrates the relationships of the pixels in a plurality offrames when the mixture ratio α is calculated.

FIG. 60 is a block diagram illustrating another configuration of themixture-ratio estimation processor 401.

FIG. 61 illustrates an exemplary estimated mixture ratio.

FIG. 62 is a flowchart illustrating the mixture-ratio estimatingprocessing by using a model corresponding to a covered background area.

FIG. 63 is a block diagram illustrating an example of the configurationof a foreground/background separator 105.

FIG. 64A illustrates an input image, a foreground component image, and abackground component image.

FIG. 64B illustrates a model of an input image, a foreground componentimage, and a background component image.

FIG. 65 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 66 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 67 illustrates a model in which pixel values are expanded in thetime direction and the period corresponding to the shutter time isdivided.

FIG. 68 is a block diagram illustrating an example of the configurationof a separating portion 601.

FIG. 69A illustrates an example of a separated foreground componentimage.

FIG. 69B illustrates an example of a separated background componentimage.

FIG. 70 is a flowchart illustrating the processing for separating aforeground and a background.

FIG. 71 is a block diagram illustrating an example of the configurationof a motion-blur adjusting unit 106.

FIG. 72 illustrates the unit of processing.

FIG. 73 illustrates a model in which the pixel values of a foregroundcomponent image are expanded in the time direction and the periodcorresponding to the shutter time is divided.

FIG. 74 illustrates a model in which the pixel values of a foregroundcomponent image are expanded in the time direction and the periodcorresponding to the shutter time is divided.

FIG. 75 illustrates a model in which the pixel values of a foregroundcomponent image are expanded in the time direction and the periodcorresponding to the shutter time is divided.

FIG. 76 illustrates a model in which the pixel values of a foregroundcomponent image are expanded in the time direction and the periodcorresponding to the shutter time is divided.

FIG. 77 illustrates another configuration of the motion-blur adjustingunit 106.

FIG. 78 is a flowchart illustrating the processing for adjusting theamount of motion blur contained in a foreground component imageperformed by the motion-blur adjusting unit 106.

FIG. 79 is a block diagram illustrating another example of theconfiguration of the motion-blur adjusting unit 106.

FIG. 80 illustrates an example of a model in which the relationshipsbetween pixel values and foreground components are indicated.

FIG. 81 illustrates the calculation of foreground components.

FIG. 82 illustrates the calculation of foreground components.

FIG. 83 is a flowchart illustrating the processing for eliminatingmotion blur contained in a foreground.

FIG. 84 is a block diagram illustrating another configuration of thefunction of the signal processing apparatus.

FIG. 85 illustrates the configuration of a synthesizer 1001.

FIG. 86 is a block diagram illustrating still another configuration ofthe function of the signal processing apparatus.

FIG. 87 is a block diagram illustrating the configuration of amixture-ratio calculator 1101.

FIG. 88 is a block diagram illustrating the configuration of aforeground/background separator 1102.

FIG. 89 is a block diagram illustrating still another configuration ofthe function of the signal processing apparatus.

FIG. 90 illustrates the configuration of a synthesizer 1201.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 illustrates an embodiment of a signal processing apparatusaccording to the present invention. A CPU (Central Processing Unit) 21executes various types of processing according to programs stored in aROM (Read Only Memory) 22 or in a storage unit 28. Programs executed bythe CPU 21 and data are stored in a RAM (Random Access Memory) 23 asrequired. The CPU 21, the ROM 22, and the RAM 23 are connected to eachother by a bus 24.

An input/output interface 25 is also connected to the CPU 21 via the bus24. An input unit 26, which is formed of a keyboard, a mouse, amicrophone, and so on, and an output unit 27, which is formed of adisplay, a speaker, and so on, are connected to the input/outputinterface 25. The CPU 21 executes various types of processing inresponse to a command input from the input unit 26. The CPU 21 thenoutputs an image or sound obtained as a result of the processing to theoutput unit 27.

The storage unit 28 connected to the input/output interface 25 is formedof, for example, a hard disk, and stores programs executed by the CPU 21and various types of data. A communication unit 29 communicates with anexternal device via the Internet or another network. In this example,the communication unit 29 serves as an obtaining unit for obtaining anoutput of a sensor.

Alternatively, a program may be obtained via the communication unit 29and stored in the storage unit 28.

A drive 30 connected to the input/output interface 25 drives a magneticdisk 51, an optical disc 52, a magneto-optical disk 53, a semiconductormemory 54, or the like, when such a recording medium is attached to thedrive 30, and obtains a program or data stored in the correspondingmedium. The obtained program or data is transferred to the storage unit28 and stored therein if necessary.

FIG. 2 is a block diagram illustrating the signal processing apparatus.

It does not matter whether the individual functions of the signalprocessing apparatus are implemented by hardware or software. That is,the block diagrams of this specification may be hardware block diagramsor software functional block diagrams.

In this specification, an image to be captured corresponding to anobject in the real world is referred to as an image object.

An input image supplied to the signal processing apparatus is suppliedto an object extracting unit 101, an area specifying unit 103, amixture-ratio calculator 104, and a foreground/background separator 105.

The object extracting unit 101 extracts a rough image objectcorresponding to a foreground object contained in the input image, andsupplies the extracted image object to a motion detector 102. The objectextracting unit 101 detects, for example, an outline of the foregroundimage object contained in the input image so as to extract a rough imageobject corresponding to the foreground object.

The object extracting unit 101 extracts a rough image objectcorresponding to a background object contained in the input image, andsupplies the extracted image object to the motion detector 102. Theobject extracting unit 101 extracts a rough image object correspondingto the background object from, for example, the difference between theinput image and the extracted image object corresponding to theforeground object.

Alternatively, for example, the object extracting unit 101 may extractthe rough image object corresponding to the foreground object and therough image object corresponding to the background object from thedifference between the background image stored in a built-in backgroundmemory and the input image.

The motion detector 102 calculates a motion vector of the roughlyextracted image object corresponding to the foreground object accordingto a technique, such as block matching, gradient, phase correlation, orpel-recursive technique, and supplies the calculated motion vector andthe motion-vector positional information (which is information forspecifying the positions of the pixels corresponding to the motionvector) to the area specifying unit 103 and a motion-blur adjusting unit106.

The motion vector output from the motion detector 102 containsinformation corresponding to the amount of movement v.

The motion detector 102 may output the motion vector of each imageobject, together with the pixel positional information for specifyingthe pixels of the image object, to the motion-blur adjusting unit 106.

The amount of movement v is a value indicating a positional change in animage corresponding to a moving object in units of the pixel pitch. Forexample, if an object image corresponding to a foreground is moving suchthat it is displayed at a position four pixels away from a referenceframe when it is positioned in the subsequent frame, the amount ofmovement v of the object image corresponding to the foreground is 4.

The object extracting unit 101 and the motion detector 102 are neededwhen adjusting the amount of motion blur corresponding to a movingobject.

The area specifying unit 103 determines to which of a foreground area, abackground area, or a mixed area each pixel of the input image belongs,and supplies information indicating to which area each pixel belongs(hereinafter referred to as “area information”) to the mixture-ratiocalculator 104, the foreground/background separator 105, and themotion-blur adjusting unit 106.

The mixture-ratio calculator 104 calculates the mixture ratiocorresponding to the pixels contained in a mixed area 63 (hereinafterreferred to as the “mixture ratio α”) based on the input image and thearea information supplied from the area specifying unit 103, andsupplies the resulting mixture ratio to the foreground/backgroundseparator 105.

The mixture ratio α is a value indicating the ratio of the imagecomponents corresponding to the background object (hereinafter alsoreferred to as “background components”) to the pixel value as expressedby equation (3), which is shown below.

The foreground/background separator 105 separates the input image into aforeground component image formed of only the image componentscorresponding to the foreground object (hereinafter also referred to as“foreground components”) and a background component image formed of onlythe background components based on the area information supplied fromthe area specifying unit 103 and the mixture ratio α supplied from themixture-ratio calculator 104, and supplies the foreground componentimage to the motion-blur adjusting unit 106 and a selector 107. Theseparated foreground component image may be set as the final output. Amore precise foreground and background can be obtained compared to aknown method in which only a foreground and a background are specifiedwithout considering the mixed area.

The motion-blur adjusting unit 106 determines the unit of processingindicating at least one pixel contained in the foreground componentimage based on the amount of movement v obtained from the motion vectorand based on the area information. The unit of processing is data thatspecifies a group of pixels to be subjected to the motion-bluradjustments.

Based on the amount by which the motion blur is to be adjusted, which isinput into the signal processing apparatus, the foreground componentimage supplied from the foreground/background separator 105, the motionvector and the positional information thereof supplied from the motiondetector 102, and the unit of processing, the motion-blur adjusting unit106 adjusts the amount of motion blur contained in the foregroundcomponent image by removing, decreasing, or increasing the motion blurcontained in the foreground component image. The motion-blur adjustingunit 106 then outputs the foreground component image in which amount ofmotion blur is adjusted to the selector 107. It is not essential thatthe motion vector and the positional information thereof be used.

Motion blur is a distortion contained in an image corresponding to amoving object caused by the movement of an object to be captured in thereal world and the image-capturing characteristics of the sensor.

The selector 107 selects one of the foreground component image suppliedfrom the foreground/background separator 105 and the foregroundcomponent image in which the amount of motion blur is adjusted suppliedfrom the motion-blur adjusting unit 106 based on, for example, aselection signal reflecting a user's selection, and outputs the selectedforeground component image.

An input image supplied to the signal processing apparatus is discussedbelow with reference to FIGS. 3 through 18.

FIG. 3 illustrates image capturing performed by a sensor. The sensor isformed of, for example, a CCD (Charge-Coupled Device) video cameraprovided with a CCD area sensor, which is a solid-state image-capturingdevice. An object 111 corresponding to a foreground in the real worldmoves, for example, horizontally from the left to the right, between anobject 112 corresponding to a background and the sensor.

The sensor captures the image of the object 111 corresponding to theforeground together with the image of the object 112 corresponding tothe background. The sensor outputs the captured image in units offrames. For example, the sensor outputs an image having 30 frames persecond. The exposure time of the sensor can be 1/30 second. The exposuretime is a period from when the sensor starts converting input light intoelectrical charge until when the conversion from the input light to theelectrical charge is finished. The exposure time is hereinafter alsoreferred to as a “shutter time”.

FIG. 4 illustrates the arrangement of pixels. In FIG. 4, symbols Athrough I indicate the individual pixels. The pixels are disposed on aplane of a corresponding image. One detection device corresponding toeach pixel is disposed on the sensor. When the sensor performs imagecapturing, each detection device outputs a pixel value of thecorresponding pixel forming the image. For example, the position of thedetection device in the X direction corresponds to the horizontaldirection on the image, while the position of the detection device inthe Y direction corresponds to the vertical direction on the image.

As shown in FIG. 5, the detection device, which is, for example, a CCD,converts input light into electrical charge during a periodcorresponding to a shutter time, and stores the converted electricalcharge. The amount of charge is almost proportional to the intensity ofthe input light and the period for which the light is input. Thedetection device sequentially adds the electrical charge converted fromthe input light to the stored electrical charge during the periodcorresponding to the shutter time. That is, the detection deviceintegrates the input light during the period corresponding to theshutter time and stores the electrical charge corresponding to theamount of integrated light. It can be considered that the detectiondevice has an integrating function with respect to time.

The electrical-charge stored in the detection device is converted into avoltage value by a circuit (not shown), and the voltage value is furtherconverted into a pixel value, such as digital data, and is output.Accordingly, each pixel value output from the sensor is a valueprojected on a linear space, which is a result of integrating a certainthree-dimensional portion of the object corresponding to the foregroundor the background with respect to the shutter time.

The signal processing apparatus extracts significant informationembedded in the output signal, for example, the mixture ratio α, by thestorage operation of the sensor. The signal processing apparatus adjuststhe amount of distortion, for example, the amount of motion blur, causedby the mixture of the foreground image object itself. The signalprocessing apparatus also adjusts the amount of distortion caused by themixture of the foreground image object and the background image object.

FIGS. 6A and 6B illustrate an image obtained by capturing a movingobject corresponding to a foreground and a stationary objectcorresponding to a background. FIG. 6A illustrates an image obtained bycapturing a moving object corresponding to a foreground and a stationaryobject corresponding to a background. In the example shown in FIG. 6A,the object corresponding to the foreground is moving horizontally fromthe left to the right with respect to the screen.

FIG. 6B illustrates a model obtained by expanding pixel valuescorresponding to one line of the image shown in FIG. 6A in the timedirection. The horizontal direction shown in FIG. 6B corresponds to thespatial direction X in FIG. 6A.

The values of the pixels in the background area are formed only from thebackground components, that is, the image components corresponding tothe background object. The values of the pixels in the foreground areaare formed only from the foreground components, that is, the imagecomponents corresponding to the foreground object.

The values of the pixels of the mixed area are formed from thebackground components and the foreground components. Since the values ofthe pixels in the mixed area are formed from the background componentsand the foreground components, it may be referred to as a “distortionarea”. The mixed area is further classified into a covered backgroundarea and an uncovered background area.

The covered background area is a mixed area at a position correspondingto the leading end in the direction in which the foreground object ismoving, where the background components are gradually covered with theforeground over time.

In contrast, the uncovered background area is a mixed area correspondingto the trailing end in the direction in which the foreground object ismoving, where the background components gradually appear over time.

As discussed above, the image containing the foreground area, thebackground area, or the covered background area or the uncoveredbackground area is input into the area specifying unit 103, themixture-ratio calculator 104, and the foreground/background separator105 as the input image.

FIG. 7 illustrates the background area, the foreground area, the mixedarea, the covered background area, and the uncovered background areadiscussed above. In the areas corresponding to the image shown in FIG.6A, the background area is a stationary portion, the foreground area isa moving portion, the covered background area of the mixed area is aportion that changes from the background to the foreground, and theuncovered background area of the mixed area is a portion that changesfrom the foreground to the background.

FIG. 8 illustrates a model obtained by expanding in the time directionthe pixel values of the pixels aligned side-by-side in the imageobtained by capturing the image of the object corresponding to thestationary foreground and the image of the object corresponding to thestationary background. For example, as the pixels aligned side-by-side,pixels arranged in one line on the screen can be selected.

The pixel values indicated by F01 through F04 shown in FIG. 8 are valuesof the pixels corresponding to the object of the stationary foreground.The pixel values indicated by B01 through B04 shown in FIG. 8 are valuesof the pixels corresponding to the object of the stationary background.

Time elapses from the top to the bottom in FIG. 8 in the verticaldirection in FIG. 8. The position at the top side of the rectangle inFIG. 8 corresponds to the time at which the sensor starts convertinginput light into electrical charge, and the position at the bottom sideof the rectangle in FIG. 8 corresponds to the time at which the sensorfinishes the conversion from the input light into the electrical charge.That is, the distance from the top side to the bottom side of therectangle in FIG. 8 corresponds to the shutter time.

The pixels shown in FIG. 8 are described below assuming that, forexample, the shutter time is equal to the frame size.

The horizontal direction in FIG. 8 corresponds to the spatial directionX in FIG. 6A. More specifically, in the example shown in FIG. 8, thedistance from the left side of the rectangle indicated by “F01” in FIG.8 to the right side of the rectangle indicated by “B04” is eight timesthe pixel pitch, i.e., eight consecutive pixels.

When the foreground object and the background object are stationary, thelight input into the sensor does not change during the periodcorresponding to the shutter time.

The period corresponding to the shutter time is divided into two or moreportions of equal periods. For example, if the number of virtual dividedportions is 4, the model shown in FIG. 8 can be represented by the modelshown in FIG. 9. The number of virtual divided portions can be setaccording to the amount of movement v of the object corresponding to theforeground within the shutter time. For example, the number of virtualdivided portions is set to 4 when the amount of movement v is 4, and theperiod corresponding to the shutter time is divided into four portions.

The uppermost line in FIG. 9 corresponds to the first divided periodfrom when the shutter has opened. The second line in FIG. 9 correspondsto the second divided period from when the shutter has opened. The thirdline in FIG. 9 corresponds to the third divided period from when theshutter has opened. The fourth line in FIG. 9 corresponds to the fourthdivided period from when the shutter has opened.

The shutter time divided in accordance with the amount of movement v isalso hereinafter referred to as the “shutter time/v”.

When the object corresponding to the foreground is stationary, the lightinput into the sensor does not change, and thus, the foregroundcomponent F01/v is equal to the value obtained by dividing the pixelvalue F01 by the number of virtual divided portions. Similarly, when theobject corresponding to the foreground is stationary, the foregroundcomponent F02/v is equal to the value obtained by dividing the pixelvalue F02 by the number of virtual divided portions, the foregroundcomponent F03/v is equal to the value obtained by dividing the pixelvalue F03 by the number of virtual divided portions, and the foregroundcomponent F04/v is equal to the value obtained by dividing the pixelvalue F04 by the number of virtual divided portions.

When the object corresponding to the background is stationary, the lightinput into the sensor does not change, and thus, the backgroundcomponent B01/v is equal to the value obtained by dividing the pixelvalue B01 by the number of virtual divided portions. Similarly, when theobject corresponding to the background is stationary, the backgroundcomponent B02/v is equal to the value obtained by dividing the pixelvalue B02 by the number of virtual divided portions, the backgroundcomponent B03/v is equal to the value obtained by dividing the pixelvalue B03 by the number of virtual divided portions, and the backgroundcomponent B04/v is equal to the value obtained by dividing the pixelvalue B04 by the number of virtual divided portions.

More specifically, when the object corresponding to the foreground isstationary, the light corresponding to the foreground object-input intothe sensor does not change during the period corresponding to theshutter time. Accordingly, the foreground component F01/v correspondingto the first portion of the shutter time/v from when the shutter hasopened, the foreground component F01/v corresponding to the secondportion of the shutter time/v from when the shutter has opened, theforeground component F01/v corresponding to the third portion of theshutter time/v from when the shutter has opened, and the foregroundcomponent F01/v corresponding to the fourth portion of the shuttertime/v from when the shutter has opened become the same value. The sameapplies to F02/v through F04/v, as in the case of F01/v.

When the object corresponding to the background is stationary, the lightcorresponding to the background object input into the sensor does notchange during the period corresponding to the shutter time. Accordingly,the background component B01/v corresponding to the first portion of theshutter time/v from when the shutter has opened, the backgroundcomponent B01/v corresponding to the second portion of the shuttertime/v from when the shutter has opened, the background component B01/vcorresponding to the third portion of the shutter time/v from when theshutter has opened, and the background component B01/v corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened become the same value. The same applies to B02/v through B04/v.

A description is given of the case in which the object corresponding tothe foreground is moving and the object corresponding to the backgroundis stationary.

FIG. 10 illustrates a model obtained by expanding in the time directionthe pixel values of the pixels in one line, including a coveredbackground area, when the object corresponding to the foreground ismoving to the right in FIG. 10. In FIG. 10, the amount of movement v is4. Since one frame is a short period, it can be assumed that the objectcorresponding to the foreground is a rigid body moving with constantvelocity. In FIG. 10, the object image corresponding to the foregroundis moving such that it is positioned four pixels to the right withrespect to a reference frame when it is displayed in the subsequentframe.

In FIG. 10, the pixels from the leftmost pixel to the fourth pixelbelong to the foreground area. In FIG. 10, the pixels from the fifthpixel to the seventh pixel from the left belong to the mixed area, whichis the covered background area. In FIG. 10, the rightmost pixel belongsto the background area.

The object corresponding to the foreground is moving such that itgradually covers the object corresponding to the background over time.Accordingly, the components contained in the pixel values of the pixelsbelonging to the covered background area change from the backgroundcomponents to the foreground components at a certain time during theperiod corresponding to the shutter time.

For example, the pixel value M surrounded by the thick frame in FIG. 10is expressed by equation (1) below.M=B02/v+B02/v+F07/v+F06/v  (1)

For example, the fifth pixel from the left contains a backgroundcomponent corresponding to one portion of the shutter time/v andforeground components corresponding to three portions of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 1/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains background components correspondingto three portions of the shutter time/v and a foreground componentcorresponding to one portion of the shutter time/v, and thus, themixture ratio α of the seventh pixel from the left is 3/4.

It can be assumed that the object corresponding to the foreground is arigid body, and the foreground object is moving with constant velocitysuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, for example, the foreground component F07/v of thefourth pixel from the left in FIG. 10 corresponding to the first portionof the shutter time/v from when the shutter has opened is equal to theforeground component of the fifth pixel from the left in FIG. 10corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F07/v is equalto the foreground component of the sixth pixel from the left in FIG. 10corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the seventh pixelfrom the left in FIG. 10 corresponding to the fourth portion of theshutter time/v from when the shutter has opened.

It can be assumed that the object corresponding to the foreground is arigid body, and the foreground object is moving with constant velocitysuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, for example, the foreground component F06/v of thethird pixel from the left in FIG. 10 corresponding to the first portionof the shutter time/v from when the shutter has opened is equal to theforeground component of the fourth pixel from the left in FIG. 10corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F06/v is equalto the foreground component of the fifth pixel from the left in FIG. 10corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the sixth pixel fromthe left in FIG. 10 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened.

It can be assumed that the object corresponding to the foreground is arigid body, and the foreground object is moving with constant velocitysuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, for example, the foreground component F05/v of thesecond pixel from the left in FIG. 10 corresponding to the first portionof the shutter time/v from when the shutter has opened is equal to theforeground component of the third pixel from the left in FIG. 10corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F05/v is equalto the foreground component of the fourth pixel from the left in FIG. 10corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the fifth pixel fromthe left in FIG. 10 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened.

It can be assumed that the object corresponding to the foreground is arigid body, and the foreground object is moving with constant velocitysuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, for example, the foreground component F04/v of theleft most pixel in FIG. 10 corresponding to the first portion of theshutter time/v from when the shutter has opened is equal to theforeground component of the second pixel from the left in FIG. 10corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F04/v is equalto the foreground component of the third pixel from the left in FIG. 10corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the fourth pixelfrom the left in FIG. 10 corresponding to the fourth portion of theshutter time/v from when the shutter has opened.

Since the foreground area corresponding to the moving object containsmotion blur as discussed above, it can also be referred to as a“distortion area”.

FIG. 11 illustrates a model obtained by expanding in the time directionthe pixel values of the pixels in one line including an uncoveredbackground area when the object corresponding to the foreground ismoving to the right in FIG. 11. In FIG. 11, the amount of movement v is4. Since one frame is a short period, it can be assumed that the objectcorresponding to the foreground is a rigid body moving with constantvelocity. In FIG. 11, the object image corresponding to the foregroundis moving to the right such that it is positioned four pixels to theright with respect to a reference frame when it is displayed in thesubsequent frame.

In FIG. 11, the pixels from the leftmost pixel to the fourth pixelbelong to the background area. In FIG. 11, the pixels from the fifthpixel to the seventh pixels from the left belong to the mixed area,which is an uncovered background area. In FIG. 11, the rightmost pixelbelongs to the foreground area.

The object corresponding to the foreground which covers the objectcorresponding to the background is moving such that it is graduallyremoved from the object corresponding to the background over time.Accordingly, the components contained in the pixel values of the pixelsbelonging to the uncovered background area change from the foregroundcomponents to the background components at a certain time of the periodcorresponding to the shutter time.

For example, the pixel value M′ surrounded by the thick frame in FIG. 11is expressed by equation (2).M′=F02/v+F01/v+B26/v+B26/v  (2)

For example, the fifth pixel from the left contains backgroundcomponents corresponding to three portions of the shutter time/v and aforeground component corresponding to one shutter portion of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 3/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains a background componentcorresponding to one portion of the shutter time/v and foregroundcomponents corresponding to three portions of the shutter time/v, andthus, the mixture ratio α of the seventh pixel from the left is 1/4.

When equations (1) and (2) are generalized, the pixel value M can beexpressed by equation (3):

$\begin{matrix}{M = {{\alpha \cdot B} + {\sum\limits_{i}^{\;}{{Fi}/v}}}} & (3)\end{matrix}$where α is the mixture ratio, B indicates a pixel value of thebackground, and Fi/v designates a foreground component.

It can be assumed that the object corresponding to the foreground is arigid body, which is moving with constant velocity, and the amount ofmovement is 4. Accordingly, for example, the foreground component F01/vof the fifth pixel from the left in FIG. 11 corresponding to the firstportion of the shutter time/v from when the shutter has opened is equalto the foreground component of the sixth pixel from the left in FIG. 11corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F01/v is equalto the foreground component of the seventh pixel from the left in FIG.11 corresponding to the third portion of the shutter time/v from whenthe shutter has opened, and the foreground component of the eighth pixelfrom the left in FIG. 11 corresponding to the fourth portion of theshutter time/v from when the shutter has opened.

It can be assumed that the object corresponding to the foreground is arigid body, which is moving with constant velocity, and the amount ofmovement v is 4. Accordingly, for example, the foreground componentF02/v of the sixth pixel from the left in FIG. 11 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the seventh pixel from the left inFIG. 11 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F02/vis equal to the foreground component of the eighth pixel from the leftin FIG. 11 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened.

It can be assumed that the object corresponding to the foreground is arigid body, which is moving with constant velocity, and the amount ofmovement v is 4. Accordingly, for example, the foreground componentF03/v of the seventh pixel from the left in FIG. 11 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the eighth pixel from the left inFIG. 11 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened.

It has been described with reference to FIGS. 9 through 11 that thenumber of virtual divided portions is 4. The number of virtual dividedportions corresponds to the amount of movement v. Generally, the amountof movement v corresponds to the moving speed of the objectcorresponding to the foreground. For example, if the objectcorresponding to the foreground is moving such that it is displayed fourpixels to the right with respect to a certain frame when it ispositioned in the subsequent frame, the amount of movement v is set to4. The number of virtual divided portions is set to 4 in accordance withthe amount of movement v. Similarly, when the object corresponding tothe foreground is moving such that it is displayed six pixels to theleft with respect to a certain frame when it is positioned in thesubsequent frame, the amount of movement v is set to 6, and the numberof virtual divided portions is set to 6.

FIGS. 12 and 13 illustrate the relationship of the foreground area, thebackground area, and the mixed area which consists of a coveredbackground or an uncovered background, which are discussed above, to theforeground components and the background components corresponding to thedivided periods of the shutter time.

FIG. 12 illustrates an example in which pixels in the foreground area,the background area, and the mixed area are extracted from an imagecontaining a foreground corresponding to an object moving in front of astationary background. In the example shown in FIG. 12, the objectcorresponding to the foreground, which is indicated by A, ishorizontally moving with respect to the screen.

Frame #n+1 is a frame subsequent to frame #n, and frame #n+2 is a framesubsequent to frame #n+1.

Pixels in the foreground area, the background area, and the mixed areaare extracted from one of frames #n through #n+2, and the amount ofmovement v is set to 4. A model obtained by expanding the pixel valuesof the extracted pixels in the time direction is shown in FIG. 13.

Since the object corresponding to the foreground is moving, the pixelvalues in the foreground area are formed of four different foregroundcomponents corresponding to the shutter time/v. For example, theleftmost pixel of the pixels in the foreground area shown in FIG. 13consists of F01/v, F02/v, F03/v, and F04/v. That is, the pixels in theforeground contain motion blur.

Since the object corresponding to the background is stationary, lightinput into the sensor corresponding to the background during the shuttertime does not change. In this case, the pixel values in the backgroundarea do not contain motion blur.

The pixel values in the mixed area consisting of a covered backgroundarea or an uncovered background area are formed of foreground componentsand background components.

A description is given below of a model obtained by expanding in thetime direction the pixel values of the pixels which are alignedside-by-side in a plurality of frames and which are located at the samepositions when the frames are overlapped when the image corresponding tothe object is moving. For example, when the image corresponding to theobject is moving horizontally with respect to the screen, pixels alignedon the screen can be selected as the pixels aligned side-by-side.

FIG. 14 illustrates a model obtained by expanding in the time directionthe pixels which are aligned side-by-side in three frames of an imageobtained by capturing an object corresponding to a stationary backgroundand which are located at the same positions when the frames areoverlapped. Frame #n is the frame subsequent to frame #n−1, and frame#n+1 is the frame subsequent to frame #n. The same applies to the otherframes.

The pixel values B01 through B12 shown in FIG. 14 are pixel valuescorresponding to the stationary background object. Since the objectcorresponding to the background is stationary, the pixel values of thecorresponding pixels in frame #n−1 through frame #n+1 do not change. Forexample, the pixel in frame #n and the pixel in frame #n+1 located atthe corresponding position of the pixel having the pixel value B05 inframe #n−1 have the pixel value B05.

FIG. 15 illustrates a model obtained by expanding in the time directionthe pixels which are aligned side-by-side in three frames of an imageobtained by capturing an object corresponding to a foreground that ismoving to the right in FIG. 15 together with an object corresponding toa stationary background and which are located at the same positions whenthe frames are overlapped. The model shown in FIG. 15 contains a coveredbackground area.

In FIG. 15, it can be assumed that the object corresponding to theforeground is a rigid body moving with constant velocity, and that it ismoving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4, and thenumber of virtual divided portions is 4.

For example, the foreground component of the leftmost pixel of frame#n−1 in FIG. 15 corresponding to the first portion of the shutter time/vfrom when the shutter has opened is F12/v, and the foreground componentof the second pixel from the left in FIG. 15 corresponding to the secondportion of the shutter time/v from when the shutter has opened is alsoF12/v. The foreground component of the third pixel from the left in FIG.15 corresponding to the third portion of the shutter time/v from whenthe shutter has opened and the foreground component of the fourth pixelfrom the left in FIG. 15 corresponding to the fourth portion of theshutter time/v from when the shutter has opened are F12/v.

The foreground component of the leftmost pixel of frame #n−1 in FIG. 15corresponding to the second portion of the shutter time/v from when theshutter has opened is F11/v. The foreground component of the secondpixel from the left in FIG. 15 corresponding to the third portion of theshutter time/v from when the shutter has opened is also F11/v. Theforeground component of the third pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened is F11/v.

The foreground component of the leftmost pixel of frame #n−1 in FIG. 15corresponding to the third portion of the shutter time/v from when theshutter has opened is F10/v. The foreground component of the secondpixel from the left in FIG. 15 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened is also F10/v. Theforeground component of the leftmost pixel of frame #n−1 in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened is F09/v.

Since the object corresponding to the background is stationary, thebackground component of the second pixel from the left of frame #n−1 inFIG. 15 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B01/v. The background components of thethird pixel from the left of frame #n−1 in FIG. 15 corresponding to thefirst and second portions of the shutter time/v from when the shutterhas opened are B02/v. The background components of the fourth pixel fromthe left of frame #n−1 in FIG. 15 corresponding to the first throughthird portions of the shutter time/v from when the shutter has openedare B03/v.

In frame #n−1 in FIG. 15, the leftmost pixel from the left belongs tothe foreground area, and the second through fourth pixels from the leftbelong to the mixed area, which is a covered background area.

The fifth through twelfth pixels from the left of frame #n−1 in FIG. 15belong to the background area, and the pixel values thereof are B04through B11, respectively.

The first through fifth pixels from the left in frame #n in FIG. 15belong to the foreground area. The foreground component in the shuttertime/v in the foreground area of frame #n is any one of F05/v throughF12/v.

It can be assumed that the object corresponding to the foreground is arigid body moving with constant velocity, and that it is moving suchthat the foreground image is displayed four pixels to the right in thesubsequent frame. Accordingly, the foreground component of the fifthpixel from the left of frame #n in FIG. 15 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the sixth pixel from the left in FIG. 15corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theseventh pixel from the left in FIG. 15 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the eighth pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

The foreground component of the fifth pixel from the left of frame #n inFIG. 15 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened is F11/v. The foreground component of thesixth pixel from the left in FIG. 15 corresponding to the third portionof the shutter time/v from when the shutter has opened is also F11/v.The foreground component of the seventh pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened is F11/v.

The foreground component of the fifth pixel from the left of frame #n inFIG. 15 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened is F10/v. The foreground component of thesixth pixel from the left in FIG. 15 corresponding to the fourth portionof the shutter time/v from when the shutter has opened is also F10/v.The foreground component of the fifth pixel from the left of frame #n inFIG. 15 corresponding to the fourth portion of the shutter time/v fromwhen the shutter has opened is F09/v.

Since the object corresponding to the background is stationary, thebackground component of the sixth pixel from the left of frame #n inFIG. 15 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B05/v. The background components of theseventh pixel from the left of frame #n in FIG. 15 corresponding to thefirst and second portions of the shutter time/v from when the shutterhas opened are B06/v. The background components of the eighth pixel fromthe left of frame #n in FIG. 15 corresponding to the first through thirdportions of the shutter time/v from when the shutter has opened areB07/v.

In frame #n in FIG. 15, the sixth through eighth pixels from the leftbelong to the mixed area, which is a covered background area.

The ninth through twelfth pixels from the left of frame #n in FIG. 15belong to the background area, and the pixel values thereof are B08through B11, respectively.

The first through ninth pixels from the left in frame #n+1 in FIG. 15belong to the foreground area. The foreground component in the shuttertime/v in the foreground area of frame #n+1 is any one of F01/v throughF12/v.

It can be assumed that the object corresponding to the foreground is arigid body moving with constant velocity, and that it is moving suchthat the foreground image is displayed four pixels to the right in thesubsequent frame. Accordingly, the foreground component of the ninthpixel from the left of frame #n+1 in FIG. 15 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the tenth pixel from the left in FIG. 15corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theeleventh pixel from the left in FIG. 15 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

The foreground component of the ninth pixel from the left of frame #n+1in FIG. 15 corresponding to the second portion of the shutter time/vfrom when the shutter has opened is F11/v. The foreground component ofthe tenth pixel from the left in FIG. 15 corresponding to the thirdportion of the shutter time/v from when the shutter has opened is alsoF11/v. The foreground component of the eleventh pixel from the left inFIG. 15 corresponding to the fourth portion of the shutter time/v fromwhen the shutter has opened is F11/v.

The foreground component of the ninth pixel from the left of frame #n+1in FIG. 15 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened is F10/v. The foreground component of thetenth pixel from the left in FIG. 15 corresponding to the fourth portionof the shutter time/v from when the shutter has opened is also F10/v.The foreground component of the ninth pixel from the left of frame #n+1in FIG. 15 corresponding to the fourth portion of the shutter time/vfrom when the shutter has opened is F09/v.

Since the object corresponding to the background is stationary, thebackground component of the tenth pixel from the left of frame #n+1 inFIG. 15 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B09/v. The background components of theeleventh pixel from the left of frame #n+1 in FIG. 15 corresponding tothe first and second portions of the shutter time/v from when theshutter has opened are B10/v. The background components of the twelfthpixel from the left of frame #n+1 in FIG. 15 corresponding to the firstthrough third portions of the shutter time/v from when the shutter hasopened are B11/v.

In frame #n+1 in FIG. 15, the tenth through twelfth pixels from the leftbelong to the mixed area, which is a covered background area.

FIG. 16 is a model of an image obtained by extracting the foregroundcomponents from the pixel values shown in FIG. 15.

FIG. 17 illustrates a model obtained by expanding in the time directionthe pixels which are aligned side-by-side in three frames of an imageobtained by capturing an object corresponding to a foreground that ismoving to the right in FIG. 17 together with an object corresponding toa stationary background and which are located at the same positions whenthe frames are overlapped. The model shown in FIG. 17 contains anuncovered background area.

In FIG. 17, it can be assumed that the object corresponding to theforeground is a rigid body moving with constant velocity, and that it ismoving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4.

For example, the foreground component of the leftmost pixel of frame#n−1 in FIG. 17 corresponding to the first portion of the shutter time/vfrom when the shutter has opened is F13/v, and the foreground componentof the second pixel from the left in FIG. 17 corresponding to the secondportion of the shutter time/v from when the shutter has opened is alsoF13/v. The foreground component of the third pixel from the left in FIG.17 corresponding to the third portion of the shutter time/v from whenthe shutter has opened and the foreground component of the fourth pixelfrom the left in FIG. 17 corresponding to the fourth portion of theshutter time/v from when the shutter has opened are F13/v.

The foreground component of the second pixel from the left of frame #n−1in FIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is F14/v. The foreground component of thethird pixel from the left in FIG. 17 corresponding to the second portionof the shutter time/v from when the shutter has opened is also F14/v.The foreground component of the third pixel from the left in FIG. 17corresponding to the first portion of the shutter time/v from when theshutter has opened is F15/v.

Since the object corresponding to the background is stationary, thebackground components of the leftmost pixel of frame #n−1 in FIG. 17corresponding to the second through fourth portions of the shuttertime/v from when the shutter has opened are B25/v. The backgroundcomponents of the second pixel from the left of frame #n−1 in FIG. 17corresponding to the third and fourth portions of the shutter time/vfrom when the shutter has opened are B26/v. The background component ofthe third pixel from the left of frame #n−1 in FIG. 17 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is B27/v.

In frame #n−1 in FIG. 17, the leftmost pixel through the third pixelbelong to the mixed area, which is an uncovered background area.

The fourth through twelfth pixels from the left of frame #n−1 in FIG. 17belong to the foreground area. The foreground component of the frame isany one of F13/v through F24/v.

The leftmost pixel through the fourth pixel from the left of frame #n inFIG. 17 belong to the background area, and the pixel values thereof areB25 through B28, respectively.

It can be assumed that the object corresponding to the foreground is arigid body moving with constant velocity, and that it is moving suchthat it is displayed four pixels to the right in the subsequent frame.Accordingly, the foreground component of the fifth pixel from the leftof frame #n in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F13/v, and the foregroundcomponent of the sixth pixel from the left in FIG. 17 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F13/v. The foreground component of the seventh pixel fromthe left in FIG. 17 corresponding to the third portion of the shuttertime/v from when the shutter has opened and the foreground component ofthe eighth pixel from the left in FIG. 17 corresponding to the fourthportion of the shutter time/v from when the shutter has opened areF13/v.

The foreground component of the sixth pixel from the left of frame #n inFIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is F14/v. The foreground component of theseventh pixel from the left in FIG. 17 corresponding to the secondportion of the shutter time/v from when the shutter has opened is alsoF14/v. The foreground component of the eighth pixel from the left inFIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is F15/v.

Since the object corresponding to the background is stationary, thebackground components of the fifth pixel from the left of frame #n inFIG. 17 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B29/v. Thebackground components of the sixth pixel from the left of frame #n inFIG. 17 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B30/v. The backgroundcomponent of the seventh pixel from the left of frame #n in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B31/v.

In frame #n in FIG. 17, the fifth pixel through the seventh pixel fromthe left belong to the mixed area, which is an uncovered backgroundarea.

The eighth through twelfth pixels from the left of frame #n in FIG. 17belong to the foreground area. The value in the foreground area of frame#n corresponding to the period of the shutter time/v is any one of F13/vthrough F20/v.

The leftmost pixel through the eighth pixel from the left of frame #n+1in FIG. 17 belong to the background area, and the pixel values thereofare B25 through B32, respectively.

It can be assumed that the object corresponding to the foreground is arigid body moving with constant velocity, and that it is moving suchthat it is displayed four pixels to the right in the subsequent frame.Accordingly, the foreground component of the ninth pixel from the leftof frame #n+1 in FIG. 17 corresponding to the first portion of theshutter time/v from when the shutter has opened is F13/v, and theforeground component of the tenth pixel from the left in FIG. 17corresponding to the second portion of the shutter time/v from when theshutter has opened is also F13/v. The foreground component of theeleventh pixel from the left in FIG. 17 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F13/v.

The foreground component of the tenth pixel from the left of frame #n+1in FIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is F14/v. The foreground component of theeleventh pixel from the left in FIG. 17 corresponding to the secondportion of the shutter time/v from when the shutter has opened is alsoF14/v. The foreground component of the twelfth pixel from the left inFIG. 17 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is F15/v.

Since the object corresponding to the background is stationary, thebackground components of the ninth pixel from the left of frame #n+1 inFIG. 17 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B33/v. Thebackground components of the tenth pixel from the left of frame #n+1 inFIG. 17 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B34/v. The backgroundcomponent of the eleventh pixel from the left of frame #n+1 in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B35/v.

In frame #n+1 in FIG. 17, the ninth through eleventh pixels from theleft in FIG. 17 belong to the mixed area, which is an uncoveredbackground area.

The twelfth pixel from the left of frame #n+1 in FIG. 17 belongs to theforeground area. The foreground component in the shutter time/v in theforeground area of frame #n+1 is any one of F13/v through F16/v.

FIG. 18 is a model of an image obtained by extracting the foregroundcomponents from the pixel values shown in FIG. 17.

Referring back to FIG. 2, the area specifying unit 103 specifies flagsindicating to which of a foreground area, a background area, a coveredbackground area, or an uncovered background area the individual pixelsof the input image belong by using the pixel values of a plurality offrames, and supplies the flags to the mixture-ratio calculator 104 andthe motion-blur adjusting unit 106 as the area information.

The mixture-ratio calculator 104 calculates the mixture ratio α for eachpixel contained in the mixed area based on the pixel values of aplurality of frames and the area information, and supplies the resultingmixture ratio α to the foreground/background separator 105.

The foreground/background separator 105 extracts the foregroundcomponent image consisting of only the foreground components based onthe pixel values of a plurality of frames, the area information, and themixture ratio α, and supplies the foreground component image to themotion-blur adjusting unit 106.

The motion-blur adjusting unit 106 adjusts the amount of motion blurcontained in the foreground component image based on the foregroundcomponent image supplied from the foreground/background separator 105,the motion vector supplied from the motion detector 102, and the areainformation supplied from the area specifying unit 103, and then outputsthe foreground component image in which motion blur is adjusted.

The processing for adjusting the amount of motion blur performed by thesignal processing apparatus is described below with reference to theflowchart of FIG. 19. In step S11, the area specifying unit 103 executesarea specifying processing, based on an input image, for generating areainformation indicating to which of a foreground area, a background area,a covered background area, or an uncovered background area each pixel ofthe input image belongs. Details of the area specifying processing aregiven below. The area specifying unit 103 supplies the generated areainformation to the mixture-ratio calculator 104.

In step S11, the area specifying unit 103 may generate, based on theinput image, area information indicating to which of the foregroundarea, the background area, or the mixed area (regardless of whether eachpixel belongs to a covered background area or an uncovered backgroundarea) each pixel of the input image belongs. In this case, theforeground/background separator 105 and the motion-blur adjusting unit106 determine based on the direction of the motion vector whether themixed area is a covered background area or an uncovered background area.For example, if the input image is disposed in the order of theforeground area, the mixed area, and the background area in thedirection of the motion vector, it is determined that the mixed area isa covered background area. If the input image is disposed in the orderof the background area, the mixed area, and the foreground area in thedirection of the motion vector, it is determined that the mixed area isan uncovered background area.

In step S12, the mixture-ratio calculator 104 calculates the mixtureratio α for each pixel contained in the mixed area based on the inputimage and the area information. Details of the mixture ratio calculatingprocessing are given below. The mixture-ratio calculator 104 suppliesthe resulting mixture ratio α to the foreground/background separator105.

In step S13, the foreground/background separator 105 extracts theforeground components from the input image based on the area informationand the mixture ratio α, and supplies the foreground components to themotion-blur adjusting unit 106 as the foreground component image.

In step S14, the motion-blur adjusting unit 106 generates, based on themotion vector and the area information, the unit of processing thatindicates the positions of consecutive pixels disposed in the movingdirection and belonging to any of the uncovered background area, theforeground area, and the covered background area, and adjusts the amountof motion blur contained in the foreground components corresponding tothe unit of processing. Details of the processing for adjusting theamount of motion blur are given below.

In step S15, the signal processing apparatus determines whether theprocessing is finished for the whole screen. If it is determined thatthe processing is not finished for the whole screen, the processproceeds to step S14, and the processing for adjusting the amount ofmotion blur for the foreground components corresponding to the unit ofprocessing is repeated.

If it is determined in step S15 that the processing is finished for thewhole screen, then the process ends.

In this manner, the signal processing apparatus is capable of adjustingthe amount of motion blur contained in the foreground by separating theforeground and the background. That is, the signal processing apparatusis capable of adjusting the amount of motion blur contained in sampleddata indicating the pixel values of the foreground pixels.

The configuration of each of the area specifying unit 103, themixture-ratio calculator 104, the foreground/background separator 105,and the motion-blur adjusting unit 106 is described below.

FIG. 20 is a block diagram illustrating the configuration of the areaspecifying unit 103. The area specifying unit 103 shown in FIG. 20 doesnot use a motion vector.

A background motion compensator 201 detects a motion of a background inan input image, and causes the input image to move in parallel accordingto the detected motion of the background. The background motioncompensator 201 supplies the input image that is moving in parallelaccording to the motion of the background to an area specifyingprocessor 202.

The image supplied to the area specifying processor 202 matches thebackground on the screen.

Based on the image supplied from the background motion compensator 201which matches the background on the screen, the area specifyingprocessor 202 generates area information indicating to which of anuncovered background area, a stationary area, a moving area, or acovered background area each pixel belongs, and outputs the generatedarea information.

FIG. 21 is a block diagram illustrating the configuration of the areaspecifying unit 103 in more detail.

The background motion compensator 201 is formed of a frame memory 221,motion-capturing portions 222-1 through 222-4, and image shift portions223-1 through 223-4.

The area specifying processor 202 is formed of a frame memory 224,stationary/moving determining portions 225-1 to 225-4, area determiningportions 226-1 to 226-3, a determining-flag-storing frame memory 227, asynthesizer 228, and a determining-flag-storing frame memory 229.

The frame memory 221 stores an input image in units of frames. When theimage to be processed is frame #n, the frame memory 221 stores frame#n−2, which is the frame two frames before frame #n, frame #n−1, whichis the frame one frame before frame #n, frame #n, frame #n+1, which isthe frame one frame after frame #n, frame #n+2, which is the frame twoframes after frame #n.

The motion-capturing portion 222-1 obtains a designated block having apredetermined number of pixels from frame #n+2 stored in the framememory 221. Based on the designated block, the motion-capturing portion222-1 retrieves an image portion which matches in pattern the designatedblock from the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-1 generates a motion vector based on theposition of the designated block in frame #n+2 and the position of theimage portion in frame #n which matches in pattern the designated block.

The motion-capturing portion 222-2 obtains a designated block having apredetermined number of pixels from frame #n+1 stored in the framememory 221. Based on the designated block, the motion-capturing portion222-2 retrieves an image portion which matches in pattern the designatedblock from the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-2 generates a motion vector based on theposition of the designated block in frame #n+1 and the position of theimage portion in frame #n which matches in pattern the designated block.

The motion-capturing portion 222-3 obtains a designated block having apredetermined number of pixels from frame #n−1 stored in the framememory 221. Based on the designated block, the motion-capturing portion222-3 retrieves an image portion which matches in pattern the designatedblock from the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-3 generates a motion vector based on theposition of the designated block in frame #n−1 and the position of theimage portion in frame #n which matches in pattern the designated block.

The motion-capturing portion 222-4 obtains a designated block having apredetermined number of pixels from frame #n−2 stored in the framememory 221. Based on the designated block, the motion-capturing portion222-4 retrieves an image portion which matches in pattern the designatedblock from the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-4 generates a motion vector based on theposition of the designated block in frame #n−2 and the position of theimage portion in frame #n which matches in pattern the designated block.

In the following description, the motion-capturing portions 222-1through 222-4 are referred to simply as a motion-capturing portion 222if it is not essential to identify them individually.

For example, as shown in FIG. 22, the motion-capturing portion 222divides one frame of image into portions consisting of m pixels by npixels, as indicated by A in FIG. 22. In each of the divided portions, adesignated block consisting of M pixels by N pixels are chosen, asindicated by B in FIG. 22.

The motion-capturing portion 222 retrieves an image portion whichmatches in pattern the designated block in each of the divided portionsfrom the corresponding frame of image so as to generate a motion vectorof each designated block. The motion-capturing portion 222 generates amotion vector corresponding to two frames based on the motion vectorgenerated for each designated block. For example, the motion-capturingportion 222 calculates average of the generated motion vectors of thedesignated blocks so as to use the resulting motion vector as a motionvector corresponding to two frames.

Generally, a background image object is larger than a foreground imageobject in an input image, and the motion-capturing portion 222 canoutput a motion vector corresponding to the motion of the backgroundimage object.

The motion-capturing portion 222 may generate a motion vectorcorresponding to the motion of the background image object by performingfull-screen block matching between two frame of images.

Alternatively, the motion-capturing portion 222 may extract thebackground image object from the input image and may generate a motionvector corresponding to the motion of the background image object basedon the extracted image object.

The image shift portion 223-1 shifts frame #n+2 stored in the framememory 221 in parallel based on the motion vector corresponding to thebackground image object supplied from the motion-capturing portion222-1, and supplies the resulting image of frame #n+2 to the framememory 224 of the area specifying processor 202.

The image shift portion 223-2 shifts frame #n+1 stored in the framememory 221 in parallel based on the motion vector corresponding to thebackground image object supplied from the motion-capturing portion222-2, and supplies the resulting image of frame #n+1 to the framememory 224 of the area specifying processor 202.

The image shift portion 223-3 shifts frame #n−1 stored in the framememory 221 in parallel based on the motion vector corresponding to thebackground image object supplied from the motion-capturing portion222-3, and supplies the resulting image of frame #n−1 to the framememory 224 of the area specifying processor 202.

The image shift portion 223-4 shifts frame #n−2 stored in the framememory 221 in parallel based on the motion vector corresponding to thebackground image object supplied from the motion-capturing portion222-4, and supplies the resulting image of frame #n−2 to the framememory 224 of the area specifying processor 202.

The frame memory 221 supplies the image of frame #n to the frame memory224 of the area specifying processor 202.

The images supplied to the frame memory 224 of the area specifyingprocessor 202 from the image shift portions 223-1 through 223-4, and theimage supplied from the frame memory 221 match the background on thescreen.

The frame memory 224 of the area specifying processor 202 stores theimages supplied from the image shift portions 223-1 through 223-4 or theimage supplied from the frame memory 221 in units of frames.

A stationary/moving determining portion 225-1 reads the pixel value ofthe pixel of frame #n+2 located at the same position as a target pixelof frame #n in which the area to which the pixel belongs is determined,and reads the pixel value of the pixel of frame #n+1 located at the sameposition of the target pixel of frame #n from the frame memory 224, andcalculates the absolute value of the difference between the read pixelvalues. The stationary/moving determining portion 225-1 determineswhether the absolute value of the difference between the pixel value offrame #n+2 and the pixel value of frame #n+1 is greater than a presetthreshold Th. If it is determined that the difference is greater thanthe threshold Th, a stationary/moving determination indicating “moving”is supplied to an area determining portion 226-1. If it is determinedthat the absolute value of the difference between the pixel value of thepixel of frame #n+2 and the pixel value of the pixel of frame #n+1 issmaller than or equal to the threshold Th, the stationary/movingdetermining portion 225-1 supplies a stationary/moving determinationindicating “stationary” to the area determining portion 226-1.

A stationary/moving determining portion 225-2 reads the pixel value of atarget pixel of frame #n in which the area to which the pixel belongs isdetermined, and reads the pixel value of the pixel of frame #n+1 locatedat the same position as the target pixel of frame #n from the framememory 224, and calculates the absolute value of the difference betweenthe pixel values. The stationary/moving determining portion 225-2determines whether the absolute value of the difference between thepixel value of frame #n+1 and the pixel value of frame #n is greaterthan a preset threshold Th. If it is determined that the absolute valueof the difference between the pixel values is greater than the thresholdTh, a stationary/moving determination indicating “moving” is supplied tothe area determining portion 226-1 and an area determining portion226-2. If it is determined that the absolute value of the differencebetween the pixel value of the pixel of frame #n+1 and the pixel valueof the pixel of frame #n is smaller than or equal to the threshold Th,the stationary/moving determining portion 225-2 supplies astationary/moving determination indicating “stationary” to the areadetermining portion 226-1 and the area determining portion 226-2.

A stationary/moving determining portion 225-3 reads the pixel value of atarget pixel of frame #n in which the area to which the pixel belongs isdetermined, and reads the pixel value of the pixel of frame #n−1 locatedat the same position as the target pixel of frame #n from the framememory 224, and calculates the absolute value of the difference betweenthe pixel values. The stationary/moving determining portion 225-3determines whether the absolute value of the difference between thepixel value of frame #n and the pixel value of frame #n−1 is greaterthan a preset threshold Th. If it is determined that the absolute valueof the difference between the pixel values is greater than the thresholdTh, a stationary/moving determination indicating “moving” is supplied tothe area determining portion 226-2 and an area determining portion226-3. If it is determined that the absolute value of the differencebetween the pixel value of the pixel of frame #n and the pixel value ofthe pixel of frame #n−1 is smaller than or equal to the threshold Th,the stationary/moving determining portion 225-3 supplies astationary/moving determination indicating “stationary” to the areadetermining portion 226-2 and the area determining portion 226-3.

A stationary/moving determining portion 225-4 reads the pixel value ofthe pixel of frame #n−1 located at the same position as a target pixelof frame #n in which the area to which the pixel belongs is determined,and reads the pixel value of the pixel of frame #n−2 located at the sameposition as the target pixel of frame #n from the frame memory 224, andcalculates the absolute value of the difference between the pixelvalues. The stationary/moving determining portion 225-4 determineswhether the absolute value of the difference between the pixel value offrame #n−1 and the pixel value of frame #n−2 is greater than a presetthreshold Th. If it is determined that the absolute value of thedifference between the pixel values is greater than the threshold Th, astationary/moving determination indicating “moving” is supplied to thearea determining portion 226-3. If it is determined that the absolutevalue of the difference between the pixel value of the pixel of frame#n−1 and the pixel value of the pixel of frame #n−2 is smaller than orequal to the threshold Th, the stationary/moving determining portion225-4 supplies a stationary/moving determination indicating “stationary”to the area determining portion 226-3.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-1 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 225-2 indicates “moving”, the areadetermining portion 226-1 determines that the target pixel of frame #nbelongs to an uncovered background area, and sets “1”, which indicatesthat the target pixel belongs to an uncovered background area, in anuncovered-background-area determining flag associated with the targetpixel.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-1 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 225-2 indicates “stationary”, the area specifyingunit 226-1 determines that the target pixel of frame #n does not belongto an uncovered background area, and sets “0”, which indicates that thetarget pixel does not belong to an uncovered background area, in theuncovered-background-area determining flag associated with the targetpixel.

The area determining portion 226-1 supplies theuncovered-background-area determining flag in which “1” or “0” is set asdiscussed above to a determining-flag-storing frame memory 227.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-2 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 225-3 indicates “stationary”, thearea determining portion 226-2 determines that the target pixel of frame#n belongs to the stationary area, and sets “1”, which indicates thatthe pixel belongs to the stationary area, in a stationary-areadetermining flag associated with the target pixel.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-2 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 225-3 indicates “moving”, the area determiningportion 226-2 determines that the target pixel of frame #n does notbelong to the stationary area, and sets “0”, which indicates that thepixel does not belong to the stationary area, in the stationary-areadetermining flag associated with the target pixel.

The area determining portion 226-2 supplies the stationary-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 227.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-2 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 225-3 indicates “moving”, the area determiningportion 226-2 determines that the target pixel of frame #n belongs tothe moving area, and sets “1”, which indicates that the target pixelbelongs to the moving area, in a moving-area determining flag associatedwith the target pixel.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-2 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 225-3 indicates “stationary”, thearea determining portion 226-2 determines that the target pixel of frame#n does not belong to the moving area, and sets “0”, which indicatesthat the pixel does not belong to the moving area, in the moving-areadetermining flag associated with the target pixel.

The area determining portion 226-2 supplies the moving-area determiningflag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 227.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-3 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 225-4 indicates “stationary”, the area determiningportion 226-3 determines that the target pixel of frame #n belongs to acovered background area, and sets “1”, which indicates that the targetpixel belongs to the covered background area, in acovered-background-area determining flag associated with the targetpixel.

When the stationary/moving determination supplied from thestationary/moving determining portion 225-3 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 225-4 indicates “moving”, the areadetermining portion 226-3 determines that the target pixel of frame #ndoes not belong to a covered background area, and sets “0”, whichindicates that the target pixel does not belong to a covered backgroundarea, in the covered-background-area determining flag associated withthe target pixel.

The area determining portion 226-3 supplies the covered-background-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 227.

The determining-flag-storing frame memory 227 thus stores theuncovered-background-area determining flag supplied from the areadetermining portion 226-1, the stationary-area determining flag suppliedfrom the area determining portion 226-2, the moving-area determiningflag supplied from the area determining portion 226-2, and thecovered-background-area determining flag supplied from the areadetermining portion 226-3.

The determining-flag-storing frame memory 227 supplies theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag stored therein to a synthesizer228. The synthesizer 228 generates area information indicating to whichof the uncovered background area, the stationary area, the moving area,or the covered background area each pixel belongs based on theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag supplied from thedetermining-flag-storing frame memory 227, and supplies the areainformation to a determining-flag-storing frame memory 229.

The determining-flag-storing frame memory 229 stores the areainformation supplied from the synthesizer 228, and also outputs the areainformation stored therein.

An example of the processing performed by the area specifying unit 103is described below with reference to FIGS. 23 through 27.

When the object corresponding to the foreground is moving, the positionof the image corresponding to the object on the screen changes in everyframe. As shown in FIG. 23, the image corresponding to the objectlocated at the position of a certain area in the image, as indicated byYn(x1,y1) in frame #n, is positioned at Yn+1(x2,y2) in frame #n+1, whichis subsequent to frame #n.

A model obtained by expanding in the time direction the pixel values ofthe pixels aligned side-by-side in the moving direction of the imagecorresponding to the foreground object is shown in FIG. 2A. For example,if the moving direction of the image corresponding to the foregroundobject is horizontal with respect to the screen, the model shown in FIG.24 is a model obtained by expanding in the time direction the pixelvalues of the pixels disposed on a line side-by-side.

In FIG. 24, the line in frame #n is equal to the line in frame #n+1.

The foreground components corresponding to the object contained in thesecond pixel to the thirteenth pixel from the left in frame #n arecontained in the sixth pixel through the seventeenth pixel from the leftin frame #n+1.

In frame #n, the pixels belonging to the covered background area are theeleventh through thirteenth pixels from the left, and the pixelsbelonging to the uncovered background area are the second through fourthpixels from the left. In frame #n+1, the pixels belonging to the coveredbackground area are the fifteenth through seventeenth pixels from theleft, and the pixels belonging to the uncovered background area are thesixth through eighth pixels from the left.

In the example shown in FIG. 24, since the foreground componentscontained in frame #n are moved by four pixels in frame #n+1, the amountof movement v is 4. The number of virtual divided portions is 4 inaccordance with the amount of movement v.

A description is now given of a change in pixel values of the pixelsbelonging to the mixed area in the frames before and after a designatedframe.

In FIG. 25, the pixels belonging to a covered background area in frame#n in which the background is stationary and the amount of movement v inthe foreground is 4 are the fifteenth through seventeenth pixels fromthe left. Since the amount of movement v is 4, the fifteenth throughseventeenth frames from the left in the previous frame #n−1 contain onlybackground components and belong to the background area. The fifteenththrough seventeenth pixels from the left in frame #n−2, which is onebefore frame #n−1, contain only background components and belong to thebackground area.

Since the object corresponding to the background is stationary, thepixel value of the fifteenth pixel from the left in frame #n−1 does notchange from the pixel value of the fifteenth pixel from the left inframe #n−2. Similarly, the pixel value of the sixteenth pixel from theleft in frame #n−1 does not change from the pixel value of the sixteenthpixel from the left in frame #n−2, and the pixel value of theseventeenth pixel from the left in frame #n−1 does not change from thepixel value of the seventeenth pixel from the left in frame #n−2.

That is, the pixels in frame #n−1 and frame #n−2 corresponding to thepixels belonging to the covered background area in frame #n consist ofonly background components, and the pixel values thereof do not change.Accordingly, the absolute value of the difference between the pixelvalues is almost 0. Thus, the stationary/moving determination made forthe pixels in frame #n−1 and frame #n−2 corresponding to the pixelsbelonging to the mixed area in frame #n by the stationary/movingdetermining portion 225-4 is “stationary”.

Since the pixels belonging to the covered background area in frame #ncontain foreground components, the pixel values thereof are differentfrom those of frame #n−1 consisting of only background components.Accordingly, the stationary/moving determination made for the pixelsbelonging to the mixed area in frame #n and the corresponding pixels inframe #n−1 by the stationary/moving determining portion 225-3 is“moving”.

When the stationary/moving determination result indicating “moving” issupplied from the stationary/moving determining portion 225-3, and whenthe stationary/moving determination result indicating “stationary” issupplied from the stationary/moving determining portion 225-4, asdiscussed above, the area determining portion 226-3 determines that thecorresponding pixels belong to a covered background area.

In FIG. 26, in frame #n in which the background is stationary and theamount of movement v in the foreground is 4, the pixels contained in anuncovered background area are the second through fourth pixels from theleft. Since the amount of movement v is 4, the second through fourthpixels from the left in the subsequent frame #n+1 contain onlybackground components and belong to the background area. In frame #n+2,which is subsequent to frame #n+1, the second through fourth pixels fromthe left contain only background components and belong to the backgroundarea.

Since the object corresponding to the background is stationary, thepixel value of the second pixel from the left in frame #n+2 does notchange from the pixel value of the second pixel from the left in frame#n+1. Similarly, the pixel value of the third pixel from the left inframe #n+2 does not change from the pixel value of the third pixel fromthe left in frame #n+1, and the pixel value of the fourth pixel from theleft in frame #n+2 does not change from the pixel value of the fourthpixel from the left in frame #n+1.

That is, the pixels in frame #n+1 and frame #n+2 corresponding to thepixels belonging to the uncovered background area in frame #n consist ofonly background components, and the pixel values thereof do not change.Accordingly, the absolute value of the difference between the pixelvalues is almost 0. Thus, the stationary/moving determination made forthe pixels in frame #n+1 and frame #n+2 corresponding to the pixelsbelonging to the mixed area in frame #n by the stationary/movingdetermining portion 225-1 is “stationary”.

Since the pixels belonging to the uncovered background area in frame #ncontain foreground components, the pixel values thereof are differentfrom those of frame #n+1 consisting of only background components.Accordingly, the stationary/moving determination made for the pixelsbelonging to the mixed area in frame #n and the corresponding pixels inframe #n+1 by the stationary/moving determining portion 225-2 is“moving”.

When the stationary/moving determination result indicating “moving” issupplied from the stationary/moving determining portion 225-2, and whenthe stationary/moving determination result indicating “stationary” issupplied from the stationary/moving determining portion 225-1, asdiscussed above, the area determining portion 226-1 determines that thecorresponding pixels belong to an uncovered background area.

FIG. 27 illustrates determination conditions for frame #n made by thearea specifying unit 103. When the determination result for the pixel inframe #n−2 located at the same image position as a pixel in frame #n tobe processed and for the pixel in frame #n−1 located at the sameposition as the pixel in frame #n is stationary, and when thedetermination result for the pixel in frame #n and the pixel in frame#n−1 located at the same image position as the pixel in frame #n ismoving, the area specifying unit 103 determines that the pixel in frame#n belongs to a covered background area.

When the determination result for the pixel in frame #n and the pixel inframe #n−1 located at the same image position as the pixel in frame #nis stationary, and when the determination result for the pixel in frame#n and the pixel in frame #n+1 located at the same image position as thepixel in frame #n is stationary, the area specifying unit 103 determinesthat the pixel in frame #n belongs to the stationary area.

When the determination result for the pixel in frame #n and the pixel inframe #n−1 located at the same image position as the pixel in frame #nis moving, and when the determination result for the pixel in frame #nand the pixel in frame #n+1 located at the same image position as thepixel in frame #n is moving, the area specifying unit 103 determinesthat the pixel in frame #n belongs to the moving area.

When the determination result for the pixel in frame #n and the pixel inframe #n+1 located at the same image position as the pixel in frame #nis moving, and when the determination result for the pixel in frame #n+1located at the same image position as the pixel in frame #n and thepixel in frame #n+2 located at the same image position as the pixel inframe #n is stationary, the area specifying unit 103 determines that thepixel in frame #n belongs to an uncovered background area.

FIGS. 28A through 28D illustrate examples of the area determinationresults obtained by the area specifying unit 103. In FIG. 28A, thepixels which are determined to belong to a covered background area areindicated in white. In FIG. 28B, the pixels which are determined tobelong to an uncovered background area are indicated in white.

In FIG. 28C, the pixels which are determined to belong to a moving areaare indicated in white. In FIG. 28D, the pixels which are determined tobelong to a stationary area are indicated in white.

FIG. 29 illustrates the area information indicating the mixed area, inthe form of an image, selected from the area information output from thedetermining-flag-storing frame memory 229. In FIG. 29, the pixels whichare determined to belong to the covered background area or the uncoveredbackground area, i.e., the pixels which are determined to belong to themixed area, are indicated in white. The area information indicating themixed area output from the determining-flag-storing frame memory 229designates the mixed area and the portions having a texture surroundedby the portions without a texture in the foreground area.

The area specifying processing performed by the area specifying unit 103is described below with reference to the flowchart of FIGS. 30 and 31.In step S201, the frame memory 221 obtains an image of frame #n−2through frame #n+2 including frame #n. The frame memory 221 suppliesframe #n to the frame memory 224.

In step S202, the motion-capturing portion 222-1 obtains a motion vectorof the background between frame #n+2 and frame #n based on the image offrame #n+2 and the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-1 supplies the motion vector to the imageshift portion 223-1.

In step S203, the image shift portion 223-1 shifts the image of frame#n+2 stored in the frame memory 221 in parallel based on the motionvector supplied from the motion-capturing portion 222-1, and suppliesthe resulting image of frame #n+2 to the frame memory 224.

In step S204, the motion-capturing portion 222-2 obtains a motion vectorof the background between frame #n+1 and frame #n based on the image offrame #n+1 and the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-2 supplies the motion vector to the imageshift portion 223-2.

In step S205, the image shift portion 223-2 shifts the image of frame#n+1 stored in the frame memory 221 in parallel based on the motionvector supplied from the motion-capturing portion 222-2, and suppliesthe resulting image of frame #n+1 to the frame memory 224.

In step S206, the motion-capturing portion 222-3 obtains a motion vectorof the background between frame #n−1 and frame #n based on the image offrame #n−1 and the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-3 supplies the motion vector to the imageshift portion 223-3.

In step S207, the image shift portion 223-3 shifts the image of frame#n−1 stored in the frame memory 221 in parallel based on the motionvector supplied from the motion-capturing portion 222-3, and suppliesthe resulting image of frame #n−1 to the frame memory 224.

In step S208, the motion-capturing portion 222-4 obtains a motion vectorof the background between frame #n−2 and frame #n based on the image offrame #n−2 and the image of frame #n stored in the frame memory 221. Themotion-capturing portion 222-4 supplies the motion vector to the imageshift portion 223-4.

In step S209, the image shift portion 223-4 shifts the image of frame#n−2 stored in the frame memory 221 in parallel based on the motionvector supplied from the motion-capturing portion 222-4, and suppliesthe resulting image of frame #n−2 to the frame memory 224.

In step S210, the stationary/moving determining portion 225-3 determineswhether the determination result for the pixel in frame #n−1 and thepixel in frame #n located at the same position is stationary. If it isdetermined that the determination result is stationary, the processproceeds to step S211 in which the stationary/moving determining portion225-2 determines whether the determination result for the pixel in frame#n and the pixel in frame #n+1 located at the same position isstationary.

If it is determined in step S211 that the determination result for thepixel in frame #n and the pixel in frame #n+1 located at the sameposition is stationary, the process proceeds to step S212. In step S204,the area determining portion 226-2 sets “1”, which indicates that thepixel to be processed belongs to the stationary area, in thestationary-area determining flag associated with the pixel to beprocessed. The area determining portion 226-2 supplies thestationary-area determining flag to the determining-flag-storing framememory 227, and the process proceeds to step S213.

If it is determined in step S210 that the determination result for thepixel in frame #n−1 and the pixel in frame #n located at the sameposition is moving, or if it is determined in step S211 that thedetermination result for the pixel in frame #n and the pixel in frame#n+1 located at the same position is moving, the pixel to be processeddoes not belong to a stationary area. Accordingly, the processing ofstep S212 is skipped, and the process proceeds to step S213.

In step S213, the stationary/moving determining portion 225-3 determineswhether the determination result for the pixel in frame #n−1 and thepixel in frame #n located at the same position is moving. If it isdetermined that the determination result is moving, the process proceedsto step S214 in which the stationary/moving determining portion 225-2determines whether the determination result for the pixel in frame #nand the pixel in frame #n+1 located at the same position is moving.

If it is determined in step S214 that the determination result for thepixel in frame #n and the pixel in frame #n+1 located at the sameposition is moving, the process proceeds to step S215. In step S215, thearea determining portion 226-2 sets “1”, which indicates that the pixelto be processed belongs to a moving area, in the moving-area determiningflag associated with the pixel to be processed. The area determiningarea 226-2 supplies the moving-area determining flag to thedetermining-flag-storing frame memory 227, and the process proceeds tostep S216.

If it is determined in step S213 that the determination result for thepixel in frame #n−1 and the pixel in frame #n located at the sameposition is stationary, or if it is determined in step S214 that thedetermination result for the pixel in frame #n and the pixel in frame#n+1 located at the same position is stationary, the pixel in frame #ndoes not belong to a moving area. Accordingly, the processing of stepS215 is skipped, and the process proceeds to step S216.

In step S216, the stationary/moving determining portion 225-4 determineswhether the determination result for the pixel in frame #n−2 and thepixel in frame #n−1 located at the same position is stationary. If it isdetermined that the determination result is stationary, the processproceeds to step S217 in which the stationary/moving determining portion225-3 determines whether the determination result for the pixel in frame#n−1 and the pixel in frame #n located at the same position is moving.

If it is determined in step S217 that the determination result for thepixel in frame #n−1 and the pixel in frame #n located at the sameposition is moving, the process proceeds to step S218. In step S218, thearea determining portion 226-3 sets “1”, which indicates that the pixelto be processed belongs to a covered background area, in thecovered-background-area determining flag associated with the pixel to beprocessed. The area determining portion 226-3 supplies thecovered-background-area determining flag to the determining-flag-storingframe memory 227, and the process proceeds to step S219.

If it is determined in step S216 that the determination result for thepixel in frame #n−2 and the pixel in frame #n−1 located at the sameposition is moving, or if it is determined in step S217 that the pixelin frame #n−1 and the pixel in frame #n located at the same position isstationary, the pixel in frame #n does not belong to a coveredbackground area. Accordingly, the processing of step S218 is skipped,and the process proceeds to step S219.

In step S219, the stationary/moving determining portion 225-2 determineswhether the determination result for the pixel in frame #n and the pixelin frame #n+1 located at the same position is moving. If it isdetermined in step S219 that the determination result is moving, theprocess proceeds to step S220 in which the stationary/moving determiningportion 225-1 determines whether the determination result for the pixelin frame #n+1 and the pixel in frame #n+2 located at the same positionis stationary.

If it is determined in step S220 that the determination result for thepixel in frame #n+1 and the pixel in frame #n+2 located at the sameposition is stationary, the process proceeds to step S221. In step S221,the area determining portion 226-1 sets “1”, which indicates that thepixel to be processed belongs to an uncovered background area, in theuncovered-background-area determining flag associated with the pixel tobe processed. The area determining portion 226-1 supplies theuncovered-background-flag determining flag to thedetermining-flag-storing frame memory 227, and the process proceeds tostep S222.

If it is determined in step S219 that the determination result for thepixel in frame #n and the pixel in frame #n+1 located at the sameposition is stationary, or if it is determined in step S220 that thedetermination result for the pixel in frame #n+1 and the pixel in frame#n+2 is moving, the pixel in frame #n does not belong to an uncoveredbackground area. Accordingly, the processing of step S221 is skipped,and the process proceeds to step S222.

In step S222, the area specifying unit 103 determines whether the areasof all the pixels in frame #n are specified. If it is determined thatthe areas of all the pixels in frame #n are not yet specified, theprocess returns to step S210, and the area specifying processing isrepeated for the remaining pixels.

If it is determined in step S222 that the areas of all the pixels inframe #n are specified, the process proceeds to step S223. In step S223,the synthesizer 228 generates area information indicating the mixed areabased on the uncovered-background-area determining flag and thecovered-background-area determining flag stored in thedetermining-flag-storing frame memory 227, and also generates areainformation indicating to which of the uncovered background area, thestationary area, the moving area, or the covered background area eachpixel belongs, and sets the generated area information in thedetermining-flag-storing frame memory 229. Then, the process ends.

As discussed above, the area specifying unit 103 is capable ofgenerating area information indicating to which of the moving area, thestationary area, the uncovered background area, or the coveredbackground area each of the pixels contained in a frame belongs. Sincethe area specifying unit 103 enables the position of the backgroundimage object to match before the area specifying processing, moreprecise area information can be generated.

The area specifying unit 103 may apply logical OR to the areainformation corresponding to the uncovered background area and the areainformation corresponding to the covered background area so as togenerate area information corresponding to the mixed area, and then maygenerate area information consisting of flags indicating to which of themoving area, the stationary area, or the mixed area the individualpixels contained in the frame belong.

When the object corresponding to the foreground has a texture, the areaspecifying unit 103 is able to specify the moving area more precisely.

The area specifying unit 103 is able to output the area informationindicating the moving area as the area information indicating theforeground area, and outputs the area information indicating thestationary area as the area information indicating the background area.

FIG. 32 is a block diagram illustrating another configuration of thearea specifying unit 103. An object extracting unit 251 extracts aforeground image object corresponding to a foreground object from aninput image. The object extracting unit 251 further sets a value inpixels other than the pixels corresponding to the foreground imageobject, which indicates the other pixels belong to a background area, soas to generate an area-specified object including the foreground imageobject and the value which indicates that the pixels belong to thebackground area, and supplies the area-specified object to a motiondetector 252 and a subtracting unit 254.

The object extracting unit 251 detects, for example, an outline of theimage object corresponding to the foreground image object contained inthe input image so as to extract an image object corresponding to theforeground object. Alternatively, for example, the object extractingunit 251 may extract an image object corresponding to the foregroundobject from the difference between the background image stored in abuilt-in background memory and the input image.

Alternatively, for example, the object extracting unit 251 may detect amotion of the input image, and set a value in stationary pixels, whichindicates that the pixels belong to a background area, so as to generatean area-specified object including foreground image object and the valuewhich indicates that the pixels belong to the background area.

The image object contained in the area-specified object which is outputfrom the object extracting unit 251 contains pixels belonging to theforeground area and pixels belonging to the mixed area.

An example of a model obtained by expanding in the time direction thepixel values of pixels aligned side-by-side in the moving direction ofan image object corresponding to a foreground object is shown in FIG.33. For example, if the moving direction of the image objectcorresponding to the foreground object is horizontal with respect to thescreen, the model shown in FIG. 33 is a model obtained by expanding thepixel values of pixels disposed side-by-side on a single line in thetime domain.

In FIG. 33, the line in frame #n is the same as the line in frame #n−1.

In frame #n, the foreground components corresponding to the foregroundobject contained in the sixth through seventeenth pixels from the leftare contained in the second through thirteenth pixels from the left inframe #n−1.

In frame #n−1, the pixels belonging to the covered background area arethe eleventh through thirteenth pixels from the left, and the pixelsbelonging to the uncovered background area are the second through fourthpixels from the left.

In frame #n, the pixels belonging to the covered background area are thefifteenth through seventeenth pixels from the left, and the pixelsbelonging to the uncovered background area are the sixth through eighthpixels from the left.

In frame #n−1, the pixels belonging to the background area are the firstpixel from the left, and the fourteenth through twenty-first pixels fromthe left.

In frame #n, the pixels belonging to the background area are the firstthrough fifth pixels from the left, and the eighteenth throughtwenty-first pixels from the left.

An example of the area-specified object extracted by the objectextracting unit 251 according to the example shown in FIG. 33 is shownin FIG. 34.

When a designated pixel of the input image is a moving pixel, that is, apixel belonging to the foreground area or a pixel belonging to the mixedarea, which contains a foreground component, the object extracting unit251 sets the pixel value of the designated pixel of the input image inthe corresponding pixel of the area-specified object without modified.

When a designated pixel of the input image is a stationary pixel, theobject extracting unit 251 sets a value in the corresponding pixel ofthe area-specified object, which indicates that the pixel belongs to thebackground area.

The area-specified object having pixels containing a foregroundcomponent or pixels in which the value which indicates that the pixelsbelong to the background area is supplied, to the motion compensator 252and the subtracting unit 254.

The motion compensator 252 compensates for the motion of thearea-specified object supplied from the object extracting unit 251 basedon the motion vector and the positional information thereof suppliedfrom the motion detector 102. FIG. 35 illustrates an example of thearea-specified object of frame #n−1 which is motion-compensated for bythe motion compensator 252. The position of pixels belonging to theforeground area of the motion-compensated area-specified object in frame#n−1 corresponds to the position of pixels belonging to the foregroundarea of the area-specified object in frame #n. Similarly, the positionof pixels belonging to the mixed area of the motion-compensatedarea-specified object in frame #n−1 corresponds to the position ofpixels belonging to the mixed area of the area-specified object in frame#n.

The motion compensator 252 supplies the motion-compensatedarea-specified objects to the frame memory 253.

The frame memory 253 stores the motion-compensated area-specifiedobjects in units of frames. When the area-specified object correspondingto the foreground object of frame #n is supplied from the objectextracting unit 251 to the subtracting unit 254, the frame memory 253supplies the motion-compensated area-specified object of frame #n−1,which is previous to frame #n, to the subtracting unit 254.

The subtracting unit 254 subtracts, from the pixel value of a pixelbelonging to the foreground area of the area-specified object of frame#n supplied from the object extracting unit 251, the pixel value of apixel at the corresponding position and belonging to the foreground areaof the area-specified object of frame #n−1 supplied from the framememory 253. Then, the subtracting unit 254 determines the framedifference between the pixels belonging to the foreground area.

The subtracting unit 254 subtracts, from the pixel value of a pixelbelonging to the mixed area of the area-specified object of frame #nsupplied from the object extracting unit 251, the pixel value of a pixelat the corresponding position and belonging to the mixed area of thearea-specified object of frame #n−1 supplied from the frame memory 253.Then, the subtracting unit 254 determines the frame difference betweenthe pixels belonging to the mixed area.

If a value which indicates that the pixel of the area-specified objectof frame #n belongs to the background area is set, the subtracting unit254 does not perform subtraction.

The subtracting unit 254 sets the frame difference between the pixelsbelonging to the foreground area or the frame difference between thepixels belonging to the mixed area in the corresponding pixel of thearea-specified object, and supplies to a threshold-value processor 255the area-specified object in which the frame difference between thepixels belonging to the foreground area, the frame difference betweenthe pixels belonging to the mixed area, or a value indicating that thepixel belongs to the background area is set.

The threshold-value processor 255 compares the input threshold Th withthe frame difference between the pixels belonging to the foreground areaor the frame difference between the pixels belonging to the mixed areawhich is set in the pixel of the area-specified object supplied from thesubtracting unit 254. If it is determined that the frame difference inthe area-specified object is greater than the threshold Th, thethreshold-value processor 255 sets a value in the corresponding pixel ofthe area-specified object, which indicates that the pixel belongs to themixed area.

If it is determined that the frame difference in the area-specifiedobject is smaller than or equal to the threshold Th, the threshold-valueprocessor 255 sets a value in the corresponding pixel of thearea-specified object, which indicates that the pixel belongs to theforeground area.

The threshold-value processor 255 outputs the area-specified object, inwhich any one of the value indicating that the pixel belongs to thebackground area, the value indicating that the pixel belongs to themixed area, and the value indicating that the pixel belongs to theforeground area is set in each pixel, to an external unit or to a timechange detector 256 as area information including information indicatingthe mixed area.

FIG. 36 illustrates an example of the processing made by thethreshold-value processor 255. As shown in FIG. 36, after subjected tomotion compensation, the position of a pixel P04 in frame #n−1 whichbelongs to the foreground area and which is formed of foregroundcomponents F01/v, F02/v, F03/v, and F04/v matches the position of apixel C04 in frame #n which belongs to the foreground area and which isformed of foreground components F01/v, F02/v, F03/v, and F04/v.

After subjected to motion compensation, the position of a pixel P05 inframe #n−1 which belongs to the foreground area and which is formed offoreground components F02/v, F03/v, F04/v, and F05/v matches theposition of a pixel C05 in frame #n which belongs to the foreground areaand which is formed of foreground components F02/v, F03/v, F04/v, andF05/v.

Similarly, after subjected to motion compensation, the positions ofpixels P06, P07, P08, and P09 in frame #n−1 which belong to theforeground area match the positions of pixels C06, C07, C08, and C09 inframe #n having the same value and belonging to the foreground area,respectively.

Since the frame difference between the pixels belonging to theforeground area is zero, the threshold-value processor 255 determinesthat the frame difference is smaller than or equal to the threshold Th,and sets a value in the pixels C04, C05, C06, C07, C08, and C09, whichindicates the pixels belong to the foreground area.

Meanwhile, after subjected to motion compensation, the position of pixelP01 in frame #n−1 belonging to the mixed area corresponds to theposition of a pixel C01 in frame #n belonging to the foreground area.Since the pixel P01 contains the background component B02/v while thepixel C01 contains the background component B06/v, the subtracting unit254 outputs the frame difference between the background component B02/vand the background component B06/v.

The background component B02/v is different from the backgroundcomponent B06/v, and the threshold-value processor 255 determines thatthe frame difference is greater than the threshold Th, and sets a valuein the pixel C01, which indicates the pixel belongs to the mixed area.

After subjected to motion compensation, the position of pixel P02 inframe #n−1 belonging to the mixed area corresponds to the position of apixel C02 in frame #n belonging to the foreground area. Since the pixelP02 contains the background component B03/v while the pixel C02 containsthe background component B07/v, the subtracting unit 254 outputs theframe difference between the background component B03/v and thebackground component B07/v.

The background component B03/v is different from the backgroundcomponent B07/v, and the threshold-value processor 255 determines thatthe frame difference is greater than the threshold Th, and sets a valuein the pixel C02, which indicates the pixel belongs to the mixed area.

Similarly, the subtracting unit 254 outputs the frame difference of thepixel C03 belonging to the mixed area between the background componentB04/v and the background component B08/v, the frame difference of thepixel C10 belonging to the mixed area between the background componentB11/v and the background component B15/v, the frame difference of thepixel C11 belonging to the mixed area between the background componentB12/v and the background component B16/v, and the frame difference ofthe pixel C12 belonging to the mixed area between the backgroundcomponent B13/v and the background component B17/v.

Thus, the threshold-value processor 255 sets a value in the pixels C03,C10, C11, and C12, which indicates that the pixels belong to the mixedarea.

Based on the area information including the information indicating themixed area supplied from the threshold-value processor 255, the timechange detector 256 further specifies a covered background area and anuncovered background area to generate area information includinginformation indicating the covered background area and the uncoveredbackground area.

FIG. 37 is a block diagram illustrating the configuration of the timechange detector 256. When determining the area of a pixel in frame #n, aframe memory 261 stores the area information including the informationindicating the mixed area with respect to frame #n−1 and frame #nsupplied from the threshold-value processor 255.

Based on the area information including the information indicating themixed area with respect to frame #n−1 and frame #n stored in the framememory 261, an area determining portion 262 determines whether eachpixel in frame #n belonging to the mixed area belongs to the coveredbackground area or the uncovered background area. Then, the areadetermining portion 262 generates area information including informationindicating the covered background area and the uncovered backgroundarea, and outputs the generated area information.

As shown in FIGS. 38 and 39, when a pixel in frame #n−1 corresponding toa pixel in frame #n belonging to the mixed area belongs to theforeground area, the pixel in frame #n belonging to the mixed areabelongs to the uncovered background area.

When a pixel in frame #n−1 corresponding to a pixel in frame #nbelonging to the mixed area belongs to the background area, the pixel inframe #n belonging to the mixed area belongs to the covered backgroundarea.

In FIG. 38, symbol A designates the uncovered background area, andsymbol B designates the foreground area. In FIG. 38, symbol C designatesthe covered background, and symbol D designates the background area.

As shown in FIG. 40, the area determining portion 262 determines that apixel in frame #n belonging to the mixed area belongs to the uncoveredbackground area when the corresponding pixel in frame #n−1 belongs tothe foreground area, and determines that a pixel in frame #n belongingto the mixed area belongs to the covered background area when thecorresponding pixel in frame #n−1 belongs to the background area.

The area specifying processing performed by the area specifying unit 103is described below with reference to the flowchart of FIG. 41. In stepS251, the object extracting unit 251 of the area specifying unit 103extracts, based on the input image, a foreground image objectcorresponding to the foreground object by, for example, detecting anoutline of the foreground image object. The object extracting unit 251further sets a value in a pixel belonging to the background area, whichindicates that the pixel belongs to the background area so as togenerate an area-specified object. The object extracting unit 251supplies the generated area-specified object to the motion compensator252 and to the subtracting unit 254.

In step S252, the motion compensator 252 compensates for the motion ofthe area-specified object supplied from the object extracting unit 251based on the motion vector and the positional information thereofsupplied from the motion detector 102. The motion compensator 252supplies the motion-compensated area-specified object to the framememory 253. The frame memory 253 stores the motion-compensatedarea-specified object, and supplies the stored area-specified object tothe subtracting unit 254.

In step S253, the subtracting unit 254 determines the difference betweenthe area-specified object of frame #n supplied from the objectextracting unit 251 and the motion-compensated area-specified object offrame #n−1 supplied from the frame memory 253, and supplies theresulting difference to the threshold-value processor 255.

In step S254, the threshold-value processor 255 detects the mixed areabased on the threshold Th from the difference between the area-specifiedobject of frame #n and the motion-compensated area-specified object offrame #n−1, and outputs the area information including the informationindicating the mixed area to an external unit or supplies the areainformation to the time change detector 256.

In step S255, the time change detector 256 detects the coveredbackground area or the uncovered background area based on the areainformation including the information indicating the mixed area so as togenerate area information including information indicating the coveredbackground area or the uncovered background area. The generated areainformation is output, and then the process ends.

The processing for detecting a covered background area or a uncoveredbackground area in a mixed area to be processed, which corresponds tostep S255 in FIG. 41, is described in detail below with reference to theflowchart of FIG. 42.

In step S261, the area determining portion 262 of the time changedetector 256 determines whether or not a pixel in the previous framecorresponding to a designated pixel belonging to the mixed area belongsto the foreground area. If it is determined that the corresponding pixelin the previous frame belongs to the foreground area, the processproceeds to step S262 in which it is determined that the designatedpixel belongs to the uncovered background area, and the process ends.

If it is determined in step S261 that the corresponding pixel in theprevious frame belongs to the background area, the process proceeds tostep S263 in which the area determining portion 262 determines that thedesignated pixel belongs to the covered background area, and the processends.

As discussed above, based on a foreground image object corresponding tothe foreground object in a frame containing a designated frame, and aforeground image object, which is subjected to motion compensation, inthe previous frame to the frame containing the designated pixel, thearea specifying unit 103 is able to specify to which of the foregroundarea, the background area, or the mixed area the designated pixelbelongs so as to generate area information corresponding to the result.

The area specifying unit 103 is further able to determine whether or nota designated pixel belonging to the mixed area belongs to the uncoveredbackground area, or whether or not the designated pixel belongs to thecovered background area based on the area information of the previousframe to a frame containing the designated pixel.

FIG. 43 is a block diagram illustrating another configuration of thearea specifying unit 103. Similar portions as those shown in FIG. 32 aredesignated with the same reference numerals, and an explanation thereofis thus omitted.

An identification unit 281 identifies a covered background area or anuncovered background area based on the motion vector and the positionalinformation thereof supplied from the motion detector 102, and the areainformation including the information indicating the mixed area suppliedfrom the threshold-value processor 255 so as to generate areainformation including information indicating the covered background areaor the uncovered background area, and outputs the generated areainformation.

FIG. 44 illustrates the determining processing performed by theidentification unit 281. In FIG. 44, symbols A and B indicate anuncovered background area and a covered background area, respectively.In FIG. 44, symbols a and b indicate designated pixels. In FIG. 44,symbol V indicates a motion vector.

When a designated pixel is positioned at the leading end in thedirection in which a foreground image object corresponding to theforeground object is moving, the designated pixel belongs to the coveredbackground area. When a designated pixel is positioned at the trailingend in the direction in which a foreground image object corresponding tothe foreground object is moving, the designated pixel belongs to theuncovered background area. Therefore, the identification unit 281determines, based on the position of a designated pixel belonging to themixed area, that the designated pixel belongs to the uncoveredbackground area when a pixel the position of which is pointed by themotion vector belongs to the foreground area. When the position of adesignated pixel belonging to the mixed area is pointed by the motionvector, the identification unit 281 determines that the designated pixelbelongs to the covered background area based on the position of apredetermined pixel belonging to the foreground area.

The processing for detecting a covered background area or an uncoveredbackground area in a mixed area to be processed, which is performed bythe identification unit 281, is described below with reference to theflowchart of FIG. 45.

In step S281, the identification unit 281 determines whether or not thedesignated pixel is positioned at the leading end in the movingdirection of the foreground object. If it is determined that thedesignated pixel is positioned at the leading end in the movingdirection of the foreground object, the process proceeds to step S282 inwhich it is determined that the designated pixel belongs to the coveredbackground area, and then the process ends.

If it is determined in step S281 that the designated pixel is notpositioned at the leading end in the moving direction of the foregroundobject, this means that the designated pixel is positioned at thetrailing end in the moving direction of the foreground object. Theprocess then proceeds to step S283 in which the identification unit 281determines that the designated pixel belongs to the uncovered backgroundarea. Then the process ends.

As discussed above, the area specifying unit is able to determine, basedon the motion vector, whether a designated pixel belonging to the mixedarea belongs to the covered background area or the uncovered backgroundarea.

FIG. 46 is a block diagram illustrating the configuration of themixture-ratio calculator 104. An estimated-mixture-ratio processor 401calculates an estimated mixture ratio for each pixel by calculating amodel of a covered background area based on the input image, andsupplies the calculated estimated mixture ratio to a mixture-ratiodetermining portion 403.

An estimated-mixture-ratio processor 402 calculates an estimated mixtureratio for each pixel by calculating a model of an uncovered backgroundarea based on the input image, and supplies the calculated estimatedmixture ratio to the mixture-ratio determining portion 403.

Since it can be assumed that the object corresponding to the foregroundis moving with constant velocity within the shutter time, the mixtureratio α of the pixels belonging to a mixed area exhibits the followingcharacteristics. That is, the mixture ratio α linearly changes accordingto the positional change in the pixels. If the positional change in thepixels is one-dimensional, a change in the mixture ratio α can berepresented linearly. If the positional change in the pixels istwo-dimensional, a change in the mixture ratio α can be represented on aplane.

Since the period of one frame is short, it can be assumed that theobject corresponding to the foreground is a rigid body moving withconstant velocity.

The gradient of the mixture ratio α is inversely proportional to theamount of movement v within the shutter time of the foreground.

An example of the ideal mixture ratio α is shown in FIG. 47. Thegradient 1 of the ideal mixture ratio α in the mixed area can berepresented by the reciprocal of the amount of movement v.

As shown in FIG. 47, the ideal mixture ratio α has the value of 1 in thebackground area, the value of 0 in the foreground area, and the value ofgreater than 0 and smaller than 1 in the mixed area.

In the example shown in FIG. 48, the pixel value C06 of the seventhpixel from the left in frame #n can be indicated by equation (4) byusing the pixel value P06 of the seventh pixel from the left in frame#n−1.

$\begin{matrix}\begin{matrix}{{C06} = {{{B06}/v} + {{B06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{P06}/v} + {{P06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{2/v} \cdot {P06}} + {\sum\limits_{i = 1}^{2}{{Fi}/v}}}}\end{matrix} & (4)\end{matrix}$

In equation (4), the pixel value C06 is indicated by a pixel value M ofthe pixel in the mixed area, while the pixel value P06 is indicated by apixel value B of the pixel in the background area. That is, the pixelvalue M of the pixel in the mixed area and the pixel value B of thepixel in the background area can be represented by equations (5) and(6), respectively.M=C06  (5)B=P06  (6)

In equation (4), 2/v corresponds to the mixture ratio α. Since theamount of movement v is 4, the mixture ratio α of the seventh pixel fromthe left in frame #n is 0.5.

As discussed above, the pixel value C in the designated frame #n isconsidered as the pixel value in the mixed area, while the pixel value Pof frame #n−1 prior to frame #n is considered as the pixel value in thebackground area. Accordingly, equation (3) indicating the mixture ratioα can be represented by equation (7):C=α·P+f  (7)where f in equation (7) indicates the sum of the foreground componentsΣ_(i)Fi/v contained in the designated pixel. The variables contained inequation (7) are two factors, i.e., the mixture ratio α and the sum f ofthe foreground components.

Similarly, a model obtained by expanding in the time direction the pixelvalues in which the amount of movement is 4 and the number of virtualdivided portions is 4 in an uncovered background area is shown in FIG.49.

As in the representation of the covered background area, in theuncovered background area, the pixel value C of the designated frame #nis considered as the pixel value in the mixed area, while the pixelvalue N of frame #n+1 subsequent to frame #n is considered as thebackground area. Accordingly, equation (3) indicating the mixture ratioα can be represented by equation (8).C=α·N+f  (8)

The embodiment has been described, assuming that the background objectis stationary. However, equations (4) through (8) can be applied to thecase in which the background object is moving by using the pixel valueof a pixel located corresponding to the amount of movement v of thebackground. It is now assumed, for example, in FIG. 48 that the amountof movement v of the object corresponding to the background is 2, andthe number of virtual divided portions is 2. In this case, when theobject corresponding to the background is moving to the right in FIG.48, the pixel value B of the pixel in the background area in equation(6) is represented by a pixel value P04.

Since equations (7) and (8) each contain two variables, the mixtureratio α cannot be determined without modifying the equations. Generally,images have strong correlation in space, in which pixels located inclose proximity with each other have substantially the same pixel value.

Since the foreground components are strongly correlated in space, amodified equation so as to derive the sum f of the foreground componentsfrom the previous or subsequent frame is used to determine the mixtureratio α.

The pixel value Mc of the seventh pixel from the left in frame #n shownin FIG. 50 can be expressed by equation (9):

$\begin{matrix}{{Mc} = {{\frac{2}{v} \cdot {B06}} + {\sum\limits_{i = 11}^{12}{{Fi}/v}}}} & (9)\end{matrix}$

In the first term of the right side of equation (9), 2/v corresponds tothe mixture ratio α. The second term of the right side of equation (9)is expressed by equation (10) using the pixel value of the subsequentframe #n+1:

$\begin{matrix}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\beta \cdot {\sum\limits_{i = 7}^{10}{{Fi}/v}}}} & (10)\end{matrix}$It is assumed that equation (11) is obtained using the spatialcorrelation of the foreground components.F=F05=F06=F07=F08=F09=F10=F11=F12  (11)

By using equation (11), equation (10) can be represented by equation(12):

$\begin{matrix}\begin{matrix}{{\sum\limits_{i = 11}^{12}{{Fi}/v}} = {\frac{2}{v} \cdot F}} \\{= {\beta \cdot \frac{4}{v} \cdot F}}\end{matrix} & (12)\end{matrix}$

Therefore, β can be expressed by equation (13):β=2/4  (13)

Generally, if it is assumed that the foreground components associatedwith the mixed area are equal as given in equation (11), equation (14)can hold true for all pixels in the mixed area because of the internalratio:β=1−α  (14)

If equation (14) holds true, then equation (7) can be expanded as givenby equation (15):

$\begin{matrix}\begin{matrix}{C = {{\alpha \cdot P} + f}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot N}}}\end{matrix} & (15)\end{matrix}$

Similarly, if equation (14) holds true, then equation (8) can beexpanded as given by equation (16):

$\begin{matrix}\begin{matrix}{C = {{\alpha \cdot N} + f}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{{Fi}/v}}}}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot P}}}\end{matrix} & (16)\end{matrix}$

In equations (15) and (16), C, N, and P are known pixel values, and theonly variable contained in equations (15) and (16) is the mixture ratioα. The relation between C, N, and P in equations (15) and (16) isillustrated in FIG. 51. Symbol C indicates the pixel value of adesignated pixel in frame #n for which the mixture ratio α isdetermined. Symbol N indicates the pixel value of a pixel in frame #n+1which corresponds to the designated pixel in the spatial direction.Symbol P indicates the pixel value of a pixel in frame #n−1 whichcorresponds to the designated pixel in the spatial direction.

Since one variable is contained in each of equations (15) and (16), thepixel values of the pixels in the three frames can be used to determinethe mixture ratio α. The correct mixture ratio α can be determined bysolving equations (15) and (16) in conditions in which the foregroundcomponents associated with the mixed area are equal, that is, in aforeground image object captured when the foreground object isstationary, the pixel values of consecutive pixels which are positionedat a boundary of the image object the direction in which the foregroundobject is moving are constant, where the number of consecutive pixels istwo times the amount of movement v.

As discussed above, the mixture ratio α of the pixel belonging to thecovered background area is determined by equation (17) while the mixtureratio α of the pixel belonging to the uncovered background area isdetermined by equation (18):α=(C−N)/(P−N)  (17)α=(C−P)/(N−P)  (18)

FIG. 52 is a block diagram illustrating the configuration of theestimated-mixture-ratio processor 401. The frame memory 421 stores aninput image in units of frames, and supplies a frame, which is one framebefore the frame input as the input image, to a frame memory 422 and amixture-ratio calculator 423.

The frame memory 422 stores the input image in units of frames, andsupplies a frame, which is one frame before the frame supplied from theframe memory 421, to the mixture-ratio calculator 423.

When frame #n+1 is input to the mixture-ratio calculator 423 as an inputimage, the frame memory 421 supplies frame #n to the mixture-ratiocalculator 423, and the frame memory 422 supplies frame #n−1 to themixture-ratio calculator 423.

The mixture-ratio calculator 423 calculates equation (17) to determinethe estimated mixture ratio of a designated pixel based on the pixelvalue C of the designated pixel in frame #n, the pixel value N of thepixel in frame #n+1 which is at a position spatially corresponding tothe designated pixel, and the pixel value P of the pixel in frame #n−1which is at a position spatially corresponding to the designated pixel,and outputs the resulting estimated mixture ratio. For example, when thebackground is stationary, the mixture-ratio calculator 423 determinesthe estimated mixture ratio of a designated pixel based on the pixelvalue C of the designated pixel in frame #n, the pixel value N of thepixel in frame #n+1 which is located at the same inter-frame position asthe designated pixel, and the pixel value P of the pixel in frame #n−1which is located at the same inter-frame position as the designatedpixel, and outputs the resulting estimated mixture ratio.

In this manner, the estimated-mixture-ratio processor 401 is able tocalculate the estimated mixture ratio based on the input image, andsupplies it to the mixture-ratio determining portion 403.

The estimated-mixture-ratio processor 402 is similar to theestimated-mixture-ratio processor 401 except that theestimated-mixture-ratio processor 402 calculates equation (18) todetermine the estimated mixture ratio of a designated pixel while theestimated-mixture-ratio processor 401 calculates equation (17) todetermine the estimated mixture ratio of a designated pixel, and anexplanation thereof is thus omitted.

FIG. 53 illustrates an exemplary estimated mixture ratio determined bythe estimated-mixture-ratio processor 401. The estimated mixture ratioshown in FIG. 53 is shown per line in the case in which the amount ofmovement v of the foreground with respect to an object which is movingwith constant velocity is 11.

As shown in FIG. 47, it is found that the estimated mixture ratiosubstantially linearly changes in the mixed area.

Turning back to FIG. 46, the mixture-ratio determining portion 403 setsthe mixture ratio α based on the area information supplied from the areaspecifying unit 103 and indicating to which of the foreground area, thebackground area, the covered background area, or the uncoveredbackground area the pixel for which the mixture ratio α is to becalculated belongs. The mixture-ratio determining portion 403 sets themixture ratio α to 0 when the corresponding pixel belongs to theforeground area, and sets the mixture ratio α to 1 when thecorresponding pixel belongs to the background area. When thecorresponding pixel belongs to the covered background area, themixture-ratio determining portion 403 sets the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 401 as the mixtureratio α. When the corresponding pixel belongs to the uncoveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 402 as the mixture ratio α. The mixture-ratio determiningportion 403 outputs the mixture ratio α which has been set based on thearea information.

FIG. 54 is a block diagram illustrating another configuration of themixture-ratio calculator 104. A selector 441 supplies a pixel belongingto the covered background area and the corresponding pixel in theprevious and subsequent frames to an estimated-mixture-ratio processor442 based on the area information supplied from the area specifying unit103. The selector 441 supplies a pixel belonging to the uncoveredbackground area and the corresponding pixel in the previous andsubsequent frames to an estimated-mixture-ratio processor 443 based onthe area information supplied from the area specifying unit 103.

The estimated-mixture-ratio processor 442 calculates equation (17) basedon the pixel value input from the selector 441 to determine theestimated mixture ratio of the designated pixel belonging to the coveredbackground area, and supplies the resulting estimated mixture ratio to aselector 444.

The estimated-mixture-ratio processor 443 calculates equation (18) basedon the pixel value input from the selector 441 to determine theestimated mixture ratio of the designated pixel belonging to theuncovered background area, and supplies the resulting estimated mixtureratio to the selector 444.

Based on the area information supplied from the area specifying unit103, the selector 444 selects the estimated mixture ratio of 0 and setsit as the mixture ratio α when the designated pixel belongs to theforeground area, and selects the estimated mixture ratio of 1 and setsit as the mixture ratio α when the designated pixel belongs to thebackground area. When the designated pixel belongs to the coveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 442 and sets it asthe mixture ratio α. When the designated pixel belongs to the uncoveredbackground area, the selector 444 selects the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 443 and sets it asthe mixture ratio α. The selector 444 then outputs the mixture ratio αwhich is set by selection based on the area information.

As discussed above, the mixture-ratio calculator 104 configured as shownin FIG. 54 is able to calculate the mixture ratio α for each pixelcontained in the image, and outputs the calculated mixture ratio α.

The calculation processing for the mixture ratio α performed by themixture-ratio calculator 104 configured as shown in FIG. 46 is discussedbelow with reference to the flowchart of FIG. 55. In step S401, themixture-ratio calculator 104 obtains area information supplied from thearea specifying unit 103. In step S402, the estimated-mixture-ratioprocessor 401 executes the processing for estimating the mixture ratioby using a model corresponding to a covered background area, andsupplies the estimated mixture ratio to the mixture-ratio determiningportion 403. Details of the processing for estimating the mixture ratioare discussed below with reference to the flowchart of FIG. 56.

In step S403, the estimated-mixture-ratio processor 402 executes theprocessing for estimating the mixture ratio by using a modelcorresponding to an uncovered background area, and supplies theestimated mixture ratio to the mixture-ratio determining portion 403.

In step S404, the mixture-ratio calculator 104 determines whether themixture ratios have been estimated for the whole frame. If it isdetermined that the mixture ratios have not yet been estimated for thewhole frame, the process returns to step S402, and the processing forestimating the mixture ratio for the subsequent pixel is executed.

If it is determined in step S404 that the mixture ratios have beenestimated for the whole frame, the process proceeds to step S405. Instep S405, the mixture-ratio determining portion 403 sets the mixtureratio α based on the area information supplied from the area specifyingunit 103 and indicating to which of the foreground area, the backgroundarea, the covered background area, or the uncovered background area thepixel for which the mixture ratio α is to be calculated belongs. Themixture-ratio determining portion 403 sets the mixture ratio α to 0 whenthe corresponding pixel belongs to the foreground area, and sets themixture ratio α to 1 when the corresponding pixel belongs to thebackground area. When the corresponding pixel belongs to the coveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401 as the mixture ratio α. When the corresponding pixelbelongs to the uncovered background area, the mixture-ratio determiningportion 403 sets the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402 as the mixture ratio α. Then theprocess ends.

As discussed above, the mixture-ratio calculator 104 is able tocalculate the mixture ratio α, which indicates a feature quantitycorresponding to each pixel, based on the area information supplied fromthe area specifying unit 103, and the input image.

The processing for calculating the mixture ratio α performed by themixture-ratio calculator 104 configured as shown in FIG. 54 is similarto that discussed with reference to the flowchart of FIG. 55, and anexplanation thereof is thus omitted.

A description is now given, with reference to the flowchart of FIG. 56,of the mixture-ratio estimating processing by using a model of thecovered background area in step S402 of FIG. 55.

In step S421, the mixture-ratio calculator 423 obtains the pixel value Cof a designated pixel in frame #n from the frame memory 421.

In step S422, the mixture-ratio calculator 423 obtains the pixel value Pof a pixel in frame #n−1 corresponding to the designated pixel from theframe memory 421.

In step S423, the mixture-ratio calculator 423 obtains the pixel value Nof a pixel in frame #n+1 corresponding to the designated pixel containedin the input image.

In step S424, the mixture-ratio calculator 423 calculates the estimatedmixture ratio based on the pixel value C of the designated pixel inframe #n, the pixel value P of the pixel in frame #n−1, and the pixelvalue N of the pixel in frame #n+1.

In step S425, the mixture-ratio calculator 423 determines whether or notthe estimated-mixture-ratio calculation processing is completed for thewhole frame. If it is determined that the estimated-mixture-ratiocalculation processing is not completed for the whole frame, the processreturns to step S421, and the processing for calculating the estimatedmixture ratio of the next pixel is repeated.

If it is determined in step S425 that the estimated-mixture-ratiocalculation processing is completed for the whole frame, the processends.

As discussed above, the estimated-mixture-ratio processor 401 is able tocalculate the estimated mixture ratio based on the input image.

The mixture-ratio estimating processing by using a model correspondingto the uncovered background area in step S403 of FIG. 55 is similar tothe processing indicated by the flowchart of FIG. 56 by using theequations corresponding to a model of the uncovered background area, andan explanation thereof is thus omitted.

The estimated-mixture-ratio processors 442 and 443 shown in FIG. 54execute similar processing to the processing illustrated in theflowchart of FIG. 56 to calculate the estimated mixture ratio, and anexplanation thereof is thus omitted.

The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described processing for calculating the mixture ratio α can beapplied even if the image corresponding to the background area containsmotion. For example, if the image corresponding to the background areais uniformly moving, the estimated-mixture-ratio processor 401 shiftsthe overall image in accordance with the motion of the background, andperforms processing in a manner similar to the case in which the objectcorresponding to the background is stationary. If the imagecorresponding to the background area contains locally different motionsof background, the estimated-mixture-ratio processor 401 selects thepixels corresponding to the motions of background as the pixelsbelonging to the mixed area, and executes the above-describedprocessing.

The mixture-ratio calculator 104 may execute the mixture-ratioestimating processing on all the pixels only by using a modelcorresponding to the covered background area, and outputs the calculatedestimated mixture ratio as the mixture ratio α. In this case, themixture ratio α indicates the ratio of the background components for thepixels belonging to the covered background area, and indicates the ratioof the foreground components for the pixels belonging to the uncoveredbackground area. Concerning the pixels belonging to the uncoveredbackground area, the absolute value of the difference between thecalculated mixture ratio α and 1 is determined, and the calculatedabsolute value is set as the mixture ratio α. Then, the signalprocessing apparatus is able to determine the mixture ratio α indicatingthe ratio of the background components for the pixels belonging to theuncovered background area.

Similarly, the mixture-ratio processor 104 may execute the mixture-ratioestimating processing on all the pixels only by using a modelcorresponding to the uncovered background area, and outputs thecalculated estimated mixture ratio as the mixture ratio α.

The mixture-ratio calculator 104 which calculates the mixture ratio α byusing the linearly changing mixture ratio α is described below.

As described above, since equations (7) and (8) each contain twovariables, the mixture ratio α cannot be determined without modifyingthe equations.

The mixture ratio α linearly changes in accordance with a change in theposition of the pixels because the object corresponding to theforeground is moving with constant velocity. By utilizing thischaracteristic, an equation in which the mixture ratio α and the sum fof the foreground components are approximated in the spatial directioncan hold true. By utilizing a plurality of sets of the pixel values ofthe pixels belonging to the mixed area and the pixel values of thepixels belonging to the background area, the equations in which themixture ratio α and the sum f of the foreground components areapproximated are solved.

When a change in the mixture ratio α is approximated as a straight line,the mixture ratio α can be expressed by equation (19).α=il+p  (19)

In equation (19), i indicates the spatial index when the position of thedesignated pixel is set to 0, 1 designates the gradient of the straightline of the mixture ratio α, and p designates the intercept of thestraight line of the mixture ratio α and also indicates the mixtureratio α of the designated pixel. In equation (19), the index i is known,and the gradient l and the intercept p are unknown.

The relationship among the index i, the gradient l, and the intercept pis shown in FIG. 57. In FIG. 57, the while dot indicates the designatedpixel, and the black dots indicate the pixels located in close proximitywith the designated pixel.

By approximating the mixture ratio α as equation (19), a plurality ofdifferent mixture ratios α for a plurality of pixels can be expressed bytwo variables. In the example shown in FIG. 57, the five mixture ratiosfor five pixels are expressed by the two variables, i.e., the gradient land the intercept p.

When the mixture ratio α is approximated in the plane shown in FIG. 58,equation (19) is expanded into the plane by considering the movement vcorresponding to the two directions, i.e., the horizontal direction andthe vertical direction of the image, and the mixture ratio α can beexpressed by equation (20). In FIG. 58, the white dot indicates thedesignated pixel.α=jm+kq+p  (20)

In equation (20), j is the index in the horizontal direction and k isthe index in the vertical direction when the position of the designatedpixel is 0. In equation (20), m designates the horizontal gradient ofthe mixture ratio α in the plane, and q indicates the vertical gradientof the mixture ratio α in the plane. In equation (20), p indicates theintercept of the mixture ratio α in the plane.

For example, in frame #n shown in FIG. 48, equations (21) through (23)can hold true for C05 through C07, respectively.C05=α05·B05/v+f05  (21)C06=α06·B06/v+f06  (22)C07=α07·B07/v+f07  (23)

Assuming that the foreground components positioned in close proximitywith each other are equal to each other, i.e., that F01 through F03 areequal, equation (24) holds true by replacing F01 through F03 by Fc.f(x)=(1−α(x))·Fc  (24)In equation (24), x indicates the position in the spatial direction.

When α(x) is replaced by equation (20), equation (24) can be expressedby equation (25)

$\begin{matrix}\begin{matrix}{{f(x)} = {\left( {1 - \left( {{jm} + {kq} + p} \right)} \right) \cdot {Fc}}} \\{= {{j \cdot \left( {{- m} \cdot {Fc}} \right)} + {k \cdot \left( {{- q} \cdot {Fc}} \right)} + \left( {\left( {1 - p} \right) \cdot {Fc}} \right)}} \\{= {{js} + {kt} + u}}\end{matrix} & (25)\end{matrix}$

In equation (25), (−m·Fc), (−q·Fc), and (1−p)·Fc are replaced, asexpressed by equations (26) through (28), respectively.s=−m·Fc  (26)t=−q·Fc  (27)u=(1−p)·Fc  (28)

In equation (25), j is the index in the horizontal direction and k isthe index in the vertical direction when the position of the designatedpixel is 0.

As discussed above, since it can be assumed that the objectcorresponding to the foreground is moving with constant velocity withinthe shutter period, and that the foreground components positioned inclose proximity with each other are uniform, the sum of the foregroundcomponents can be approximated by equation (25).

When the mixture ratio α is approximated by a straight line, the sum ofthe foreground components can be expressed by equation (29).f(x)=is+u  (29)

By replacing the mixture ratio α and the sum of the foregroundcomponents in equation (9) by using equations (20) and (25), the pixelvalue M can be expressed by equation (30)

$\begin{matrix}\begin{matrix}{M = {{\left( {{jm} + {kq} + p} \right) \cdot B} + {js} + {kt} + u}} \\{= {{{jB} \cdot m} + {{kB} \cdot q} + {j \cdot s} + {k \cdot t} + u}}\end{matrix} & (30)\end{matrix}$

In equation (30), unknown variables are six factors, such as thehorizontal gradient m of the mixture ratio α in the plane, the verticalgradient q of the mixture ratio α in the plane, and the intercepts ofthe mixture ratio α in the plane, p, s, t, and u.

More specifically, according to the pixels in close proximity with thedesignated pixel, the pixel value M or the pixel value B is set in thenormal equation given in equation (30). Then, a plurality of normalequations in which the pixel value M or the pixel value B is set aresolved by the method of least squares, thereby calculating the mixtureratio α.

For example, the horizontal index j of the designated pixel is set to 0,and the vertical index k is set to 0. Then, the pixel value M or thepixel value B is set in the normal equation given in equation (30) for3×3 pixels located close to the designated pixel, thereby obtainingequations (31) through (39).M _(−1,−1)=(−1)·B _(−1,−1) ·m+(−1)·B _(−1,−1) ·q+B _(−1,−1)·p+(−1)·s+(−1)·t+u  (31)M _(0,−1)=(0)·B _(0,−1) ·m+(−1)·B _(0,−1) ·q+B _(0,−1)·p+(0)·s+(−1)·t+u  (32)M _(+1,−1)=(+1)·B _(+1,−1) m+(−1)·B _(+1,−1) ·q+B _(+1,−1)·p+(+1)·s+(−1)·t+u  (33)M _(−1,0)=(−1)·B _(−1,0) ·m+(0)·B _(−1,0) ·q+B _(−1,0)·p+(−1)·s+(0)·t+u  (34)M _(0,0)=(0)·B _(0,0) ·m+(0)·B _(0,0) ·q+B _(0,0) ·p+(0)·s+(0)·t+u  (35)M _(+1,0)=(+1)·B _(+1,0) ·m+(0)·B _(+1,0) ·q+B _(+1,0)·p+(+1)·s+(0)·t+u  (36)M _(−1,+1)=(−1)·B _(−1,+1) ·m+(+1)·B _(−1,+1) ·q+B _(−1,+1)·p+(0)·s+(+1)·t+u  (37)M _(0,+1)=(0)·B _(0,+1) ·m+(+1)·B _(0,+1) ·q+B _(0,+1)·p+(−1)·s+(+1)·t+u  (38)M+1,+1=(+1)·B+ _(1,+1) ·m+(+1)·B _(+1,+1) ·q+B _(+1,+1) ·p+(+1)·s+(+1)·t+u  (39)

Since the horizontal index j of the designated pixel is 0, and thevertical index k of the designated pixel is 0, the mixture ratio α ofthe designated pixel is equal to the value when j is 0 and k is 0 inequation (20), i.e., the mixture ratio α is equal to the intercept p inequation (20).

Accordingly, based on nine equations, i.e., equations (31) through (39),the horizontal gradient m, the vertical gradient q, and the interceptsp, s, t, and u are calculated by the method of least squares, and theintercept p is output as the mixture ratio α.

A specific process for calculating the mixture ratio α by applying themethod of least squares is as follows.

When the index i and the index k are expressed by a single index x, therelationship among the index i, the index k, and the index x can beexpressed by equation (40).x=(j+1)·3+(k+1)  (40)

It is now assumed that the horizontal gradient m, the vertical gradientq, and the intercepts p, s, t, and u are expressed by variables w0, w1,w2, w3, w4, and w5, respectively, and jB, kB, B, j, k and l areexpressed by a0, a1, a2, a3, a4, and a5, respectively. In considerationof the error ex, equations (31) through (39) can be modified intoequation (41).

$\begin{matrix}{{Mx} = {{\sum\limits_{y = 0}^{5}{{ay} \cdot {wy}}} + {ex}}} & (41)\end{matrix}$In equation (41), x is any one of the integers from 0 to 8.

Equation (42) can be found from equation (41).

$\begin{matrix}{{e\; x} = {{M\; x} - {\sum\limits_{y = 0}^{5}\;{a\;{y \cdot w}\; y}}}} & (42)\end{matrix}$

Since the method of least squares is applied, the square sum E of theerror is defined as follows, as expressed by equation (43).

$\begin{matrix}{E = {\sum\limits_{x = 0}^{8}\;{e\; x^{2}}}} & (43)\end{matrix}$

In order to minimize the error, the partial differential value of thevariable Wv with respect to the square sum E of the error should be 0. vis any one of the integers from 0 to 5. Thus, wy is determined so thatequation (44) is satisfied.

$\begin{matrix}\begin{matrix}{\frac{\partial E}{{\partial w}\; v} = {2 \cdot {\sum\limits_{x = 0}^{8}\;{e\;{x \cdot \frac{{\partial e}\; x}{{\partial w}\; v}}}}}} \\{= {{2 \cdot {\sum\limits_{x = 0}^{8}\;{e\;{x \cdot a}\; v}}} = 0}}\end{matrix} & (44)\end{matrix}$

By substituting equation (42) into equation (44), equation (45) isobtained.

$\begin{matrix}{{\sum\limits_{x = 0}^{8}\;\left( {a\;{v \cdot {\sum\limits_{y = 0}^{5}\;{a\;{y \cdot w}\; y}}}} \right)} = {\sum\limits_{x = 0}^{8}{a\;{v \cdot M}\; x}}} & (45)\end{matrix}$

For example, the sweep-out method (Gauss-Jordan elimination) is appliedto six equations obtained by substituting one of the integers from 0 to5 into v in equation (45), thereby obtaining wy. As stated above, w0 isthe horizontal gradient m, w1 is the vertical gradient q, w2 is theintercept p, w3 is s, w4 is t, and w5 is u.

As discussed above, by applying the method of least squares to theequations in which the pixel value M and the pixel value B are set, thehorizontal gradient m, the vertical gradient q, and the intercepts p, s,t, and u can be determined.

A description has been given with reference to equations (31) through(39), by assuming that the pixel value of the pixel contained in themixed area is M, and the pixel value of the pixel contained in thebackground area is B. In this case, it is necessary to set normalequations for each of the cases where the designated pixel is containedin the covered background area, or the designated pixel is contained inthe uncovered background area.

For example, if the mixture ratio α of the pixel contained in thecovered background area in frame #n shown in FIG. 48 is determined, C04through C08 of the pixels in frame #n and the pixel values P04 throughP08 of the pixels in frame #n−1 are set in the normal equations.

If the mixture ratio α of the pixels contained in the uncoveredbackground area in frame #n shown in FIG. 49 is determined, C28 throughC32 of the pixels in frame #n and the pixel values N28 through N32 ofthe pixels in frame #n+1 are set in the normal equations.

Moreover, if, for example, the mixture ratio α of the pixel contained inthe covered background area shown in FIG. 59 is calculated, thefollowing equations (46) through (54) are set. In FIG. 59, the whitedots indicate pixels considered to belong to the background, and theblack dots indicate pixels considered to belong to the mixed area. Thepixel value of the pixel for which the mixture ratio α is calculated isMc5.Mc1=(−1)·Bc1·m+(−1)·Bc1·q+Bc1·p+(−1)·s+(−1)·t+u  (46)Mc2=(0)·Bc2·m+(−1)·Bc2·q+Bc2·p+(0)·s+(−1)·t+u  (47)Mc3=(+1)·Bc3·m+(−1)·Bc3·q+Bc3·p+(+1)·s+(−1)·t+u  (48)Mc4=(−1)·Bc4·m+(0)·Bc4·q+Bc4·p+(−1)·s+(0)·t+u  (49)Mc5=(0)·Bc5·m+(0)·Bc5·q+Bc5·p+(0)·s+(0)·t+u  (50)Mc6=(+1)·Bc6·m+(0)·Bc6·q+Bc6·p+(+1)·s+(0)·t+u  (51)Mc7=(−1)·Bc7·m+(+1)·Bc7·q+Bc7·p+(−1)·s+(+1)·t+u  (52)Mc8=(0)·Bc8·m+(+1)·Bc8·q+Bc8·p+(0)·s+(+1)·t+u  (53)Mc9=(+1)·Bc9·m+(+1)·Bc9·q+Bc9·p+(+1)·s+(+1)·t+u  (54)

For calculating the mixture ratio α of the pixel contained in thecovered background area in frame #n, the pixel values Bc1 through Bc9 ofthe pixels of the background area in frame #n−1 in equations (46)through (54), respectively, corresponding to the pixels in frame #n areused.

When calculating the mixture ratio α of the pixel contained in theuncovered background area shown in FIG. 59, the following equations (55)through (63) can hold true. The pixel value of the pixel for which themixture ratio α is calculated is Mu5.Mu1=(−1)·Bu1·m+(−1)·Bu1·q+Bu1·p+(−1)·s+(−1)·t+u  (55)Mu2=(0)·Bu2·m+(−1)·Bu2·q+Bu2·p+(0)·s+(−1)·t+u  (56)Mu3=(+1)·Bu3·m+(−1)·Bu3·q+Bu3·p+(+1)·s+(−1)·t+u  (57)Mu4=(−1)·Bu4·m+(0)·Bu4·q+Bu4·p+(−1)·s+(0)·t+u  (58)Mu5=(0)·Bu5·m+(0)·Bu5·q+Bu5·p+(0)·s+(0)·t+u  (59)Mu6=(+1)·Bu6·m+(0)·Bu6·q+Bu6·p+(+1)·s+(0)·t+u  (60)Mu7=(−1)·Bu7·m+(+1)·Bu7·q+Bu7·p+(−1)·s+(+1)·t+u  (61)Mu8=(0)·Bu819 m+(+1)·Bu8·q+Bu8·p+(0)·s+(+1)·t+u  (62)Mu9=(+1)·Bu9·m+(+1)·Bu9·q+Bu9·p+(+1)·s+(+1)·t+u  (63)

For calculating the mixture ratio α of the pixel contained in theuncovered background area in frame #n, the pixel values Bu1 through Bu9of the pixels of the background area in frame #n+1 in equations (55)through (63), respectively, corresponding to the pixels in frame #n areused.

FIG. 60 is a block diagram illustrating the configuration of theestimated-mixture-ratio processor 401. An image input into theestimated-mixture-ratio processor 401 is supplied to a delay circuit 501and an adder 502.

The delay circuit 501 delays the input image for one frame, and suppliesthe image to the adder 502. When frame #n is supplied as the input imageto the adder 502, the delay circuit 501 supplies frame #n−1 to the adder502.

The adder 502 sets the pixel value of the pixel adjacent to the pixelfor which the mixture ratio α is calculated, and the pixel value offrame #n−1 in the normal equation. For example, the adder 502 sets thepixel values Mc1 through Mc9 and the pixel values Bc1 through Bc9 in thenormal equations based on equations (46) through (54), respectively. Theadder 502 supplies the normal equations in which the pixel values areset to a calculator 503.

The calculator 503 determines the estimated mixture ratio by solving thenormal equations supplied from the adder 502 using the sweep-out methodor the like, and outputs the determined estimated mixture ratio.

In this manner, the estimated-mixture-ratio processor 401 is able tocalculate the estimated mixture ratio based on the input image, andsupplies it to the mixture-ratio determining portion 403.

The estimated-mixture-ratio processor 402 is configured similarly to theestimated-mixture-ratio processor 401, and an explanation thereof isthus omitted.

FIG. 61 illustrates an example of the estimated mixture ratio calculatedby the estimated-mixture-ratio processor 401. The estimated mixtureratio shown in FIG. 61 is shown per line in the case in which the amountof movement v of the foreground with respect to an object which ismoving with constant velocity is 11 and in which the equation is set inunits of a block of 7×7 pixels.

As shown in FIG. 60, it is found that the estimated mixture ratiosubstantially linearly changes in the mixed area.

A description is now given, with reference to the flowchart of FIG. 62,of the mixture-ratio estimating processing by theestimated-mixture-ratio processor 401 having the configuration shown inFIG. 60 by using a model of the covered background area.

In step S521, the adder 502 sets the pixel value contained in the inputimage and the pixel value contained in the image supplied from the delaycircuit 501 in a normal equation corresponding to a model of the coveredbackground area.

In step S522, the estimated-mixture-ratio processor 401 determineswhether the setting of the target pixels is finished. If it isdetermined that the setting of the target pixels is not finished, theprocess returns to step S521, and the processing for setting the pixelvalues in the normal equation is repeated.

If it is determined in step S522 that the setting for the target pixelsis finished, the process proceeds to step S523. In step S523, thecalculator 503 calculates the estimated mixture ratio based on thenormal equations in which the pixels values are set, and outputs thecalculated mixture ratio.

As discussed above, the estimated-mixture-ratio processor 401 having theconfiguration shown in FIG. 60 is able to calculate the estimatedmixture ratio based on the input image.

The mixture-ratio estimating processing by using a model correspondingto the uncovered background area is similar to the processing indicatedby the flowchart of FIG. 62 by using the normal equations correspondingto a model of the uncovered background area, and an explanation thereofis thus omitted.

The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described mixture-ratio calculation processing can be applied evenif the image corresponding to the background area contains motion. Forexample, if the image corresponding to the background area is uniformlymoving, the estimated-mixture-ratio processor 401 shifts the overallimage in accordance with this motion, and performs processing in amanner similar to the case in which the object corresponding to thebackground is stationary. If the image corresponding to the backgroundarea contains locally different motions, the estimated-mixture-ratioprocessor 401 selects the pixels corresponding to the motions as thepixels belonging to the mixed area, and executes the above-describedprocessing.

As described above, the mixture-ratio calculator 102 is able tocalculate the mixture ratio α, which is a feature quantity correspondingto each pixel, based on the input image and the area informationsupplied to the area specifying unit 101.

By utilizing the mixture ratio α, it is possible to separate theforeground components and the background components contained in thepixel values while maintaining the information of motion blur containedin the image corresponding to the moving object.

By combining the images based on the mixture ratio α, it is alsopossible to create an image which contains correct motion blur thatcoincides with the speed of a moving object and which faithfullyreflects the real world.

The foreground/background separator 105 is discussed below. FIG. 63 is ablock diagram illustrating an example of the configuration of theforeground/background separator 105. The input image supplied to theforeground/background separator 105 is supplied to a separating portion601, a switch 602, and a switch 604. The area information supplied fromthe area specifying unit 103 and indicating the information of thecovered background area and the uncovered background area is supplied tothe separating portion 601. The area information indicating theforeground area is supplied to the switch 602. The area informationindicating the background area supplied to the switch 604.

The mixture ratio α supplied from the mixture-ratio calculator 104 issupplied to the separating portion 601.

The separating portion 601 separates the foreground components from theinput image based on the area information indicating the coveredbackground area, the area information indicating the uncoveredbackground area, and the mixture ratio α, and supplies the separatedforeground components to a synthesizer 603. The separating portion 601also separates the background components from the input image, andsupplies the separated background components to a synthesizer 605.

The switch 602 is closed when a pixel corresponding to the foreground isinput based on the area information indicating the foreground area, andsupplies only the pixels corresponding to the foreground contained inthe input image to the synthesizer 603.

The switch 604 is closed when a pixel corresponding to the background isinput based on the area information indicating the background area, andsupplies only the pixels corresponding to the background contained inthe input image to the synthesizer 605.

The synthesizer 603 synthesizes a foreground component image based onthe foreground components supplied from the separating portion 601 andthe pixels corresponding to the foreground supplied from the switch 602,and outputs the synthesized foreground component image. Since theforeground area and the mixed area do not overlap, the synthesizer 603applies, for example, logical OR to the foreground components and theforeground pixels, thereby synthesizing the foreground component image.

In the initializing processing executed at the start of the synthesizingprocessing for the foreground component image, the synthesizer 603stores an image whose pixel values are all 0 in a built-in frame memory.Then in the synthesizing processing for the foreground component image,the synthesizer 603 stores the foreground component image (overwritesthe previous image by the foreground component image). Accordingly, 0 isstored in the pixels corresponding to the background area in theforeground component image output from the synthesizer 603.

The synthesizer 605 synthesizes a background component image based onthe background components supplied from the separating portion 601 andthe pixels corresponding to the background supplied from the switch 604,and outputs the synthesized background component image. Since thebackground area and the mixed area do not overlap, the synthesizer 605applies, for example, logical OR to the background components and thebackground pixels, thereby synthesizing the background component image.

In the initializing processing executed at the start of the synthesizingprocessing for the background component image, the synthesizer 605stores an image whose pixel values are all 0 in a built-in frame memory.Then, in the synthesizing processing for the background component image,the synthesizer 605 stores the background component image (overwritesthe previous image by the background component image). Accordingly, 0 isstored in the pixels corresponding to the foreground area in thebackground component image output from the synthesizer 605.

FIG. 64A illustrates the input image input into theforeground/background separator 105 and the foreground component imageand the background component image output from the foreground/backgroundseparator 105. FIG. 64B illustrates a model of the input image inputinto the foreground/background separator 105 and the foregroundcomponent image and the background component image output from theforeground/background separator 105.

FIG. 64A is a schematic diagram illustrating the image to be displayed,and FIG. 64B is a model obtained by expanding in the time direction thepixels disposed in one line including the pixels belonging to theforeground area, the pixels belonging to the background area, and thepixels belonging to the mixed area corresponding to FIG. 64A.

As shown in FIGS. 64A and 64B, the background component image outputfrom the foreground/background separator 105 consists of the pixelsbelonging to the background area and the background components containedin the pixels of the mixed area.

As shown in FIGS. 64A and 64B, the foreground component image outputfrom the foreground/background separator 105 consists of the pixelbelonging to the foreground area and the foreground components containedin the pixels of the mixed area.

The pixel values of the pixels in the mixed area are separated into thebackground components and the foreground components by theforeground/background separator 105. The separated background componentsform the background component image together with the pixels belongingto the background area. The separated foreground components form theforeground component image together with the pixels belonging to theforeground area.

As discussed above, in the foreground component image, the pixel valuesof the pixels corresponding to the background area are set to 0, andsignificant pixel values are set in the pixels corresponding to theforeground area and the pixels corresponding to the mixed area.Similarly, in the background component image, the pixel values of thepixels corresponding to the foreground area are set to 0, andsignificant pixel values are set in the pixels corresponding to thebackground area and the pixels corresponding to the mixed area.

A description is given below of the processing executed by theseparating portion 601 for separating the foreground components and thebackground components from the pixels belonging to the mixed area.

FIG. 65 illustrates a model of an image indicating foreground componentsand background components in two frames including a foreground objectmoving from the left to the right in FIG. 65. In the model of the imageshown in FIG. 65, the amount of movement v is 4, and the number ofvirtual divided portions is 4.

In frame #n, the leftmost pixel and the fourteenth through eighteenthpixels from the left consist of only the background components andbelong to the background area. In frame #n, the second through fourthpixels from the left contain the background components and theforeground components, and belong to the uncovered background area. Inframe #n, the eleventh through thirteenth pixels from the left containbackground components and foreground components, and belong to thecovered background area. In frame #n, the fifth through tenth pixelsfrom the left consist of only the foreground components, and belong tothe foreground area.

In frame #n+1, the first through fifth pixels from the left and theeighteenth pixel from the left consist of only the backgroundcomponents, and belong to the background area. In frame #n+1, the sixththrough eighth pixels from the left contain background components andforeground components, and belong to the uncovered background area. Inframe #n+1, the fifteenth through seventeenth pixels from the leftcontain background components and foreground components, and belong tothe covered background area. In frame #n+1, the ninth through fourteenthpixels from the left consist of only the foreground components, andbelong to the foreground area.

FIG. 66 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the covered background area. InFIG. 66, α1 through α18 indicate mixture ratios of the individual pixelsof frame #n. In FIG. 66, the fifteenth through seventeenth pixels fromthe left belong to the covered background area.

The pixel value C15 of the fifteenth pixel from the left in frame #n canbe expressed by equation (64):

$\begin{matrix}\begin{matrix}{{C15} = {{{B15}/v} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {B15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {P15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}}\end{matrix} & (64)\end{matrix}$where α15 indicates the mixture ratio of the fifteenth pixel from theleft in frame #n, and P15 designates the pixel value of the fifteenthpixel from the left in frame #n−1.

The sum f15 of the foreground components of the fifteenth pixel from theleft in frame #n can be expressed by equation (65) based on equation(64).

$\begin{matrix}\begin{matrix}{{f15} = {{{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{C15} - {{\alpha 15} \cdot {P15}}}}\end{matrix} & (65)\end{matrix}$

Similarly, the sum f16 of the foreground components of the sixteenthpixel from the left in frame #n can be expressed by equation (66), andthe sum f17 of the foreground components of the seventeenth pixel fromthe left in frame #n can be expressed by equation (67).f16=C16−α16·P16  (66)f17=C17−α17·P17  (67)

In this manner, the foreground components fc contained in the pixelvalue C of the pixel belonging to the covered background area can beexpressed by equation (68):fc=C−α·P  (68)where P designates the pixel value of the corresponding pixel in theprevious frame.

FIG. 67 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the uncovered background area.In FIG. 67, α1 through α18 indicate mixture ratios of the individualpixels of frame #n. In FIG. 67, the second through fourth pixels fromthe left belong to the uncovered background area.

The pixel value C02 of the second pixel from the left in frame #n can beexpressed by equation (69)

$\begin{matrix}\begin{matrix}{{C02} = {{{B02}/v} + {{B02}/v} + {{B02}/v} + {{F01}/v}}} \\{= {{{\alpha 2} \cdot {B02}} + {{F01}/v}}} \\{= {{{\alpha 2} \cdot {N02}} + {{F01}/v}}}\end{matrix} & (69)\end{matrix}$where a2 indicates the mixture ratio of the second pixel from the leftin frame #n, and N02 designates the pixel value of the second pixel fromthe left in frame #n+1.

The sum f02 of the foreground components of the second pixel from theleft in frame #n can be expressed by equation (70) based on equation(69).

$\begin{matrix}\begin{matrix}{{f02} = {{F01}/v}} \\{= {{C02} - {{\alpha 2} \cdot {N02}}}}\end{matrix} & (70)\end{matrix}$

Similarly, the sum f03 of the foreground components of the third pixelfrom the left in frame #n can be expressed by equation (71), and the sumf04 of the foreground components of the fourth pixel from the left inframe #n can be expressed by equation (72).f03=C03−α3·N03  (71)f04=C04−α4·N04  (72)

In this manner, the foreground components fu contained in the pixelvalue C of the pixel belonging to the uncovered background area can beexpressed by equation (73)fu=C−αN  (73)where N designates the pixel value of the corresponding pixel in thesubsequent frame.

As discussed above, the separating portion 601 is able to separate theforeground components from the pixels belonging to the mixed area andthe background components from the pixels belonging to the mixed areabased on the information indicating the covered background area and theinformation indicating the uncovered background area contained in thearea information, and the mixture ratio α for each pixel.

FIG. 68 is a block diagram illustrating an example of the configurationof the separating portion 601 for executing the above-describedprocessing. An image input into the separating portion 601 is suppliedto a frame memory 621, and the area information indicating the coveredbackground area and the uncovered background area supplied from themixture-ratio calculator 104 and the mixture ratio α are supplied to aseparation processing block 622.

The frame memory 621 stores the input images in units of frames. When aframe to be processed is frame #n, the frame memory 621 stores frame#n−1, which is the frame one frame before frame #n, frame #n, and frame#n+1, which is the frame one frame after frame #n.

The frame memory 621 supplies the corresponding pixels in frame #n−1,frame #n, and frame #n+1 to the separation processing block 622.

The separation processing block 622 applies the calculations discussedwith reference to FIGS. 66 and 67 to the pixel values of thecorresponding pixels in frame #n−1, frame #n, and frame #n+1 suppliedfrom the frame memory 621 based on the area information indicating thecovered background area and the uncovered background area and themixture ratio α so as to separate the foreground components and thebackground components from the pixels belonging to the mixed area inframe #n, and supplies them to a frame memory 623.

The separation processing block 622 is formed of an uncovered areaprocessor 631, a covered area processor 632, a synthesizer 633, and asynthesizer 634.

A multiplier 641 of the uncovered area processor 631 multiplies thepixel value of the pixel in frame #n+1 supplied from the frame memory621 by the mixture ratio α, and outputs the resulting pixel value to aswitch 642. The switch 642 is closed when the pixel of frame #n(corresponding to the pixel in frame #n+1) supplied from the framememory 621 belongs to the uncovered background area, and supplies thepixel value multiplied by the mixture ratio α supplied from themultiplier 641 to a calculator 643 and the synthesizer 634. The valueobtained by multiplying the pixel value of the pixel in frame #n+1 bythe mixture ratio α output from the switch 642 is equivalent to thebackground components of the pixel value of the corresponding pixel inframe #n.

The calculator 643 subtracts the background components supplied from theswitch 642 from the pixel value of the pixel in frame #n supplied fromthe frame memory 621 so as to obtain the foreground components. Thecalculator 643 supplies the foreground components of the pixel in frame#n belonging to the uncovered background area to the synthesizer 633.

A multiplier 651 of the covered area processor 632 multiplies the pixelvalue of the pixel in frame #n−1 supplied from the frame memory 621 bythe mixture ratio α, and outputs the resulting pixel value to a switch652. The switch 652 is closed when the pixel of frame #n (correspondingto the pixel in frame #n−1) supplied from the frame memory 621 belongsto the covered background area, and supplies the pixel value multipliedby the mixture ratio α supplied from the multiplier 651 to a calculator653 and the synthesizer 634. The value obtained by multiplying the pixelvalue of the pixel in frame #n−1 by the mixture ratio α output from theswitch 652 is equivalent to the background components of the pixel valueof the corresponding pixel in frame #n.

The calculator 653 subtracts the background components supplied from theswitch 652 from the pixel value of the pixel in frame #n supplied fromthe frame memory 621 so as to obtain the foreground components. Thecalculator 653 supplies the foreground components of the pixel in frame#n belonging to the covered background area to the synthesizer 633.

The synthesizer 633 combines the foreground components of the pixelsbelonging to the uncovered background area and supplied from thecalculator 643 with the foreground components of the pixels belonging tothe covered background area and supplied from the calculator 653, andsupplies the synthesized foreground components to the frame memory 623.

The synthesizer 634 combines the background components of the pixelsbelonging to the uncovered background area and supplied from the switch642 with the background components of the pixels belonging to thecovered background area and supplied from the switch 652, and suppliesthe synthesized background components to the frame memory 623.

The frame memory 623 stores the foreground components and the backgroundcomponents of the pixels in the mixed area of frame #n supplied from theseparation processing block 622.

The frame memory 623 outputs the stored foreground components of thepixels in the mixed area in frame #n and the stored backgroundcomponents of the pixels in the mixed area in frame #n.

By utilizing the mixture ratio α, which indicates the feature quantity,the foreground components and the background components contained in thepixel values can be completely separated.

The synthesizer 603 combines the foreground components of the pixels inthe mixed area in frame #n output from the separating portion 601 withthe pixels belonging to the foreground area so as to generate aforeground component image. The synthesizer 605 combines the backgroundcomponents of the pixels in the mixed area in frame #n output from theseparating portion 601 with the pixels belonging to the background areaso as to generate a background component image.

FIG. 69A illustrates an example of the foreground component imagecorresponding to frame #n in FIG. 65. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thepixel values are set to 0.

The second and fourth pixels from the left belong to the uncoveredbackground area before the foreground and the background are separated.Accordingly, the background components are set to 0, and the foregroundcomponents are maintained. The eleventh through thirteenth pixels fromthe left belong to the covered background area before the foreground andthe background are separated. Accordingly, the background components areset to 0, and the foreground components are maintained. The fifththrough tenth pixels from the left consist of only the foregroundcomponents, which are thus maintained.

FIG. 69B illustrates an example of the background component imagecorresponding to frame #n in FIG. 65. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thebackground components are maintained.

The second through fourth pixels from the left belong to the uncoveredbackground area before the foreground and the background are separated.Accordingly, the foreground components are set to 0, and the backgroundcomponents are maintained. The eleventh through thirteenth pixels fromthe left belong to the covered background area before the foreground andthe background are separated. Accordingly, the foreground components areset to 0, and the background components are maintained. The fifththrough tenth pixels from the left consist of only the foregroundcomponents, and thus, the pixel values are set to 0.

The processing for separating the foreground and the background executedby the foreground/background separator 105 is described below withreference to the flowchart of FIG. 70. In step S601, the frame memory621 of the separating portion 601 obtains an input image, and storesframe #n for which the foreground and the background are separatedtogether with the previous frame #n−1 and the subsequent frame #n+1.

In step S602, the separation processing block 622 of the separatingportion 601 obtains area information supplied from the mixture-ratiocalculator 104. In step S603, the separation processing block 622 of theseparating portion 601 obtains the mixture ratio α supplied from themixture-ratio calculator 104.

In step S604, the uncovered area processor 631 extracts the backgroundcomponents from the pixel values of the pixels belonging to theuncovered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

In step S605, the uncovered area processor 631 extracts the foregroundcomponents from the pixel values of the pixels belonging to theuncovered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

In step S606, the covered area processor 632 extracts the backgroundcomponents from the pixel values of the pixels belonging to the coveredbackground area supplied from the frame memory 621 based on the areainformation and the mixture ratio α.

In step S607, the covered area processor 632 extracts the foregroundcomponents from the pixel values of the pixels belonging to the coveredbackground area supplied from the frame memory 621 based on the areainformation and the mixture ratio α.

In step S608, the synthesizer 633 combines the foreground components ofthe pixels belonging to the uncovered background area extracted in theprocessing of step S605 with the foreground components of the pixelsbelonging to the covered background area extracted in the processing ofstep S607. The synthesized foreground components are supplied to thesynthesizer 603. The synthesizer 603 further combines the pixelsbelonging to the foreground area supplied via the switch 602 with theforeground components supplied from the separating portion 601 so as togenerate a foreground component image.

In step S609, the synthesizer 634 combines the background components ofthe pixels belonging to the uncovered background area extracted in theprocessing of step S604 with the background components of the pixelsbelonging to the covered background area extracted in the processing ofstep S606. The synthesized background components are supplied to thesynthesizer 605. The synthesizer 605 further combines the pixelsbelonging to the background area supplied via the switch 604 with thebackground components supplied from the separating portion 601 so as togenerate a background component image.

In step S610, the synthesizer 603 outputs the foreground componentimage. In step S611, the synthesizer 605 outputs the backgroundcomponent image. Then the process ends.

As discussed above, the foreground/background separator 105 is able toseparate the foreground components and the background components fromthe input image based on the area information and the mixture ratio α,and outputs the foreground component image consisting of only theforeground components and the background component image consisting ofonly the background components.

Adjustments of the amount of motion blur from a foreground componentimage are described below.

FIG. 71 is a block diagram illustrating an example of the configurationof the motion-blur adjusting unit 106. The motion vector and thepositional information thereof supplied from the motion detector 102 andthe area information supplied from the area specifying unit 103 aresupplied to a unit-of-processing determining portion 801 and amodel-forming portion 802. The area information supplied from theforeground/background separator 105 is supplied to the adder 804.

The unit-of-processing determining portion 801 supplies, together withthe motion vector, the unit of processing that is generated based on themotion vector and the positional information thereof and the areainformation to the model-forming portion 802. The unit-of-processingdetermining portion 801 supplies the generated unit of processing to theadder 804.

As indicated as an example by A in FIG. 72, for example, the unit ofprocessing generated by the unit-of-processing determining portion 801indicates consecutive pixels disposed in the moving direction startingfrom the pixel corresponding to the covered background area of theforeground component image until the pixel corresponding to theuncovered background area, or indicates consecutive pixels disposed inthe moving direction starting from the pixel corresponding to theuncovered background area until the pixel corresponding to the coveredbackground area. The unit of processing is formed of two pieces of datawhich indicate, for example, the upper left point (which is the positionof the leftmost or the topmost pixel in the image designated by the unitof processing) and the lower right point.

The model-forming portion 802 forms a model based on the motion vectorand the input unit of processing. More specifically, for example, themodel-forming portion 802 may store in advance a plurality of models inaccordance with the number of pixels contained in the unit ofprocessing, the number of virtual divided portions of the pixel value inthe time direction, and the number of foreground components for eachpixel. The model-forming portion 902 then may select the model in whichthe correlation between the pixel values and the foreground componentsis designated, such as that in FIG. 73, based on the unit of processingand the number of virtual divided portions of the pixel value in thetime direction.

It is now assumed, for example, that the number of pixels correspondingto the unit of processing is 12, and that the amount of movement vwithin the shutter time is 5. Then, the model-forming portion 802 setsthe number of virtual divided portions to 5, and selects a model formedof eight types of foreground components so that the leftmost pixelcontains one foreground component, the second pixel from the leftcontains two foreground components, the third pixel from the leftcontains three foreground components, the fourth pixel from the leftcontains four pixel components, the fifth pixel from the left containsfive foreground components, the sixth pixel from the left contains fiveforeground components, the seventh pixel from the left contains fiveforeground components, the eighth pixel from the left contains fiveforeground components, the ninth pixel from the left contains fourforeground components, the tenth pixel from the left contains threeforeground components, the eleventh pixel from the left contains twoforeground components, and the twelfth pixel from the left contains oneforeground component.

Instead of selecting a model from the prestored models, themodel-forming portion 802 may generate a model based on the motionvector and the unit of processing when the motion vector and the unit ofprocessing are supplied.

The model-forming portion 802 supplies the selected model to an equationgenerator 803.

The equation generator 803 generates an equation based on the modelsupplied from the model-forming portion 802. A description is givenbelow, with reference to the model of the foreground component imageshown in FIG. 73, of equations generated by the equation generator 803when the number of foreground components is 8, the number of pixelscorresponding to the unit of processing is 12, and the amount ofmovement v is 5.

When the foreground components contained in the foreground componentimage corresponding to the shutter time/v are F01/v through F08/v, therelationships between F01/v through F08/v and the pixel values C01through C12 can be expressed by equations (74) through (85).C01=F01/v  (74)C02=F02/v+F01/v  (75)C03=F03/v+F02/v+F01v  (76)C04=F04/v+F03/v+F02/v+F01v  (77)C05=F05/v+F04/v+F03/v+F02/v+F01v  (78)C06=F06/v+F05/v+F04/v+F03/v+F02/v  (79)C07=F07/v+F06/v+F05/v+F04/v+F03/v  (80)C08=F08/v+F07/v+F06/v+F05/v+F04/v  (81)C09=F08/v+F07/v+F06/v+F05/v  (82)C10=F08/v+F07/v+F06/v  (83)C11=F08/v+F07/v  (84)C12=F08/v  (85)

The equation generator 803 generates an equation by modifying thegenerated equations. The equations generated by the equation generator803 are indicated by equations (86) though (97)C01=1·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (86)C02=1··F01/v+1·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (87)C03=1·F01/v+1·F02/v+1·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (88)C04=1·F01/v+1·F02/v+1·F03/v+1·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (89)C05=1F10/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+0·F06/v+0·F07/v+0·F08/v  (90)C06=0·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+0·F07/v+0·F08/v  (91)C07=0·F01/v+0·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+0·F08/v  (92)C08=0·F01/v+0·F02/v+0·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (93)C09=0·F01/v+0·F02/v+0·F03/v+0·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (94)C10=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+1·F06/v+1·F07/v+1·F08/v  (95)C11=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+1F07/v+1·F08/v  (96)C12=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+·F06/v+0·F07/v+1·F08/v  (97)

Equations (86) through (97) can be expressed by equation (98).

$\begin{matrix}{{C\; j} = {\sum\limits_{i = 01}^{08}{a\; i\;{j \cdot F}\;{j/v}}}} & (98)\end{matrix}$In equation (98), j designates the position of the pixel. In thisexample, j has one of the values from 1 to 12. In equation (98), idesignates the position of the foreground value. In this example, i hasone of the values from 1 to 8. In equation (98), aij has the value 0 or1 according to the values of i and j.

Equation (98) can be expressed by equation (99) in consideration of theerror.

$\begin{matrix}{{C\; j} = {{\sum\limits_{i = 01}^{08}{a\; i\;{j \cdot F}\;{j/v}}} + {e\; j}}} & (99)\end{matrix}$In equation (99), ej designates the error contained in the designatedpixel Cj.

Equation (99) can be modified into equation (100).

$\begin{matrix}{{ej} = {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {{Fi}/v}}}}} & (100)\end{matrix}$

In order to apply the method of least squares, the square sum E of theerror is defined as equation (101).

$\begin{matrix}{E = {\sum\limits_{j = 01}^{12}{ej}^{2}}} & (101)\end{matrix}$

In order to minimize the error, the partial differential value using thevariable Fk with respect to the square sum E of the error should be 0.Fk is determined so that equation (102) is satisfied.

$\begin{matrix}\begin{matrix}{\frac{\partial E}{\partial{Fk}} = {2 \cdot {\sum\limits_{j = 01}^{12}{{ej} \cdot \frac{\partial{ej}}{\partial{Fk}}}}}} \\{= {2 \cdot {\sum\limits_{j = 01}^{12}\left\{ {{\left( {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {{Fi}/v}}}} \right) \cdot \left( {{- {akj}}/v} \right)} = 0} \right.}}}\end{matrix} & (102)\end{matrix}$

In equation (102), since the amount of movement v is a fixed value,equation (103) can be deduced.

$\begin{matrix}{{\sum\limits_{j = 01}^{12}{{akj} \cdot \left( {{Cj} - {\sum\limits_{i = 01}^{08}{{aij} \cdot {{Fi}/v}}}} \right)}} = 0} & (103)\end{matrix}$

To expand equation (103) and transpose the terms, equation (104) can beobtained.

$\begin{matrix}{{\sum\limits_{j = 01}^{12}\left( {{akj} \cdot {\sum\limits_{i = 01}^{08}{{aij} \cdot {Fi}}}} \right)} = {v\;{\sum\limits_{j = 01}^{12}{{akj} \cdot {Cj}}}}} & (104)\end{matrix}$

Equation (104) is expanded into eight equations by substituting theindividual integers from 1 to 8 into k in equation (104). The obtainedeight equations can be expressed by one matrix equation. This equationis referred to as a “normal equation”.

An example of the normal equation generated by the equation generator803 based on the method of least squares is indicated by equation (105).

$\begin{matrix}{{\begin{bmatrix}5 & 4 & 3 & 2 & 1 & 0 & 0 & 0 \\4 & 5 & 4 & 3 & 2 & 1 & 0 & 0 \\3 & 4 & 5 & 4 & 3 & 2 & 1 & 0 \\2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \\1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 \\0 & 1 & 2 & 3 & 4 & 5 & 4 & 3 \\0 & 0 & 1 & 2 & 3 & 4 & 5 & 4 \\0 & 0 & 0 & 1 & 2 & 3 & 4 & 5\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05} \\{F06} \\{F07} \\{F08}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 08}^{12}{Ci}} \\{\sum\limits_{i = 07}^{11}{Ci}} \\{\sum\limits_{i = 06}^{10}{Ci}} \\{\sum\limits_{i = 05}^{09}{Ci}} \\{\sum\limits_{i = 04}^{08}{Ci}} \\{\sum\limits_{i = 03}^{07}{Ci}} \\{\sum\limits_{i = 02}^{06}{Ci}} \\{\sum\limits_{i = 01}^{05}{Ci}}\end{bmatrix}}} & (105)\end{matrix}$

When equation (105) is expressed by A·F=v·C, C, A, and v are known, andF is unknown. A and v are known when the model is formed, while Cbecomes known when the pixel value is input in the addition processing.

By calculating the foreground components according to the normalequation based on the method of least squares, the error contained inthe pixel C can be distributed.

The equation generator 803 supplies the normal equation generated asdiscussed above to the adder 804.

The adder 804 sets, based on the unit of processing supplied from theunit-of-processing determining portion 801, the pixel value C containedin the foreground component image in the matrix equation supplied fromthe equation generator 803. The adder 804 supplies the matrix in whichthe pixel value C is set to a calculator 805.

The calculator 805 calculates the foreground component Fi/v from whichmotion blur is eliminated by the processing based on a solution, such asa sweep-out method (Gauss-Jordan elimination), so as to obtain Ficorresponding to i indicating one of the integers from 0 to 8, which isthe pixel value from which motion blur is eliminated. The calculator 805then outputs the foreground component image consisting of the pixelvalues Fi without motion blur, such as that in FIG. 74, to a motion-bluradder 806 and a selector 807.

In the foreground component image without motion blur shown in FIG. 74,the reason for setting F01 through F08 in C03 through C10, respectively,is not to change the position of the foreground component image withrespect to the screen. However, F01 through F08 may be set in anydesired positions.

The motion-blur adder 806 is able to adjust the amount of motion blur byadding the amount v′ by which motion blur is adjusted, which isdifferent from the amount of movement v, for example, the amount v′ bywhich motion blur is adjusted, which is one half the value of the amountof movement v, or the amount v′ by which motion blur is adjusted, whichis irrelevant to the amount of movement v. For example, as shown in FIG.75, the motion-blur adder 806 divides the foreground pixel value Fiwithout motion blur by the amount v′ by which motion blur is adjusted soas to obtain the foreground component Fi/v′. The motion-blur adder 806then calculates the sum of the foreground components Fi/v′, therebygenerating the pixel value in which the amount of motion blur isadjusted. For example, when the amount v′ by which motion blur isadjusted is 3, the pixel value C02 is set to (F01)/v′, the pixel valueC3 is set to (F01+F02)/v′, the pixel value C04 is set to(F01+F02+F03)/v′, and the pixel value C05 is set to (F02+F03+F04)/v′.

The motion-blur adder 806 supplies the foreground component image inwhich the amount of motion blur is adjusted to a selector 807.

The selector 807 selects, based on a selection signal reflecting auser's selection, one of the foreground component image supplied fromthe calculator 805 from which motion blur is eliminated and theforeground component image supplied from the motion-blur adder 806 inwhich the amount of motion blur is adjusted, and outputs the selectedforeground component image.

As discussed above, the motion-blur adjusting unit 106 is able to adjustthe amount of motion blur based on the selection signal and the amountv′ by which motion blur is adjusted.

Also, for example, when the number of pixels corresponding to the unitof processing is 8, and the amount of movement v is 4, as shown in FIG.76, the motion-blur adjusting unit 106 generates a matrix equationexpressed by equation (106).

$\begin{matrix}{{\begin{bmatrix}4 & 3 & 2 & 1 & 0 \\3 & 4 & 3 & 2 & 1 \\2 & 3 & 4 & 3 & 2 \\1 & 2 & 3 & 4 & 3 \\0 & 1 & 2 & 3 & 4\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 05}^{08}{Ci}} \\{\sum\limits_{i = 04}^{07}{Ci}} \\{\sum\limits_{i = 03}^{06}{Ci}} \\{\sum\limits_{i = 02}^{05}{Ci}} \\{\sum\limits_{i = 01}^{04}{Ci}}\end{bmatrix}}} & (106)\end{matrix}$

In this manner, the motion-blur adjusting unit 106 calculates Fi, whichis the pixel value in which the amount of motion blur is adjusted, bysetting up the equation in accordance with the length of the unit ofprocessing. Similarly, for example, when the number of pixels containedin the unit of processing is 100, the equation corresponding to 100pixels is generated so as to calculate Fi.

FIG. 77 illustrates an example of another configuration of themotion-blur adjusting unit 106. The same elements as those shown in FIG.71 are designated with like reference numerals and an explanationthereof is thus omitted.

Based on a selection signal, a selector 821 directly supplies an inputmotion vector and a positional signal thereof to the unit-of-processingdetermining portion 801 and the model-forming portion 802.Alternatively, the selector 821 may substitute the magnitude of themotion vector by the amount v′ by which motion blur is adjusted, andthen supplies the motion vector and the positional signal thereof to theunit-of-processing determining portion 801 and the model-forming unit802.

With this arrangement, the unit-of-processing determining portion 801through the calculator 805 of the motion-blur adjusting unit 106 shownin FIG. 77 are able to adjust the amount of motion blur in accordancewith the amount of movement v and the amount v′ by which motion blur isadjusted. For example, when the amount of movement is 5, and the amountv′ by which motion blur is adjusted is 3, the unit-of-processingdetermining portion 801 through the calculator 805 of the motion-bluradjusting unit 106 shown in FIG. 77 execute computation on theforeground component image in which the amount of movement v is 5 shownin FIG. 73 according to the model shown in FIG. 75 in which the amountv′ by which motion blur is adjusted is 3. As a result, the imagecontaining motion blur having the amount of movement v of (amount ofmovement v)/(amount v′ by which motion blur is adjusted)=5/3, i.e.,about 1.7 is obtained. In this case, the calculated image does notcontain motion blur corresponding to the amount of movement v of 3.Accordingly, it should be noted that the relationship between the amountof movement v and the amount v′ by which motion blur is adjusted isdifferent from the result of the motion-blur adder 806.

As discussed above, the motion-blur adjusting unit 106 generates theequation in accordance with the amount of movement v and the unit ofprocessing, and sets the pixel values of the foreground component imagein the generated equation, thereby calculating the foreground componentimage in which the amount of motion blur is adjusted.

The processing for adjusting the amount of motion blur contained in theforeground component image executed by the motion-blur adjusting unit106 is described below with reference to the flowchart of FIG. 78.

In step S801, the unit-of-processing determining portion 801 of themotion-blur adjusting unit 106 generates the unit of processing based onthe motion vector and the area information, and supplies the generatedunit of processing to the model-forming portion 802.

In step S802, the model-forming portion 802 of the motion-blur adjustingunit 106 selects or generates the model in accordance with the amount ofmovement v and the unit of processing. In step S803, the equationgenerator 803 generates the normal equation based on the selected model.

In step S804, the adder 804 sets the pixel values of the foregroundcomponent image in the generated normal equation. In step S805, theadder 804 determines whether the pixel values of all the pixelscorresponding to the unit of processing are set. If it is determinedthat the pixel values of all the pixels corresponding to the unit ofprocessing are not yet set, the process returns to step S804, and theprocessing for setting the pixel values in the normal equation isrepeated.

If it is determined in step S805 that the pixel values of all the pixelscorresponding to the unit of processing are set, the process proceeds tostep S806. In step S806, the calculator 805 calculates the pixel valuesof the foreground in which the amount of motion blur is adjusted basedon the normal equation in which the pixel values supplied from the adder804 are set. Then the process ends.

As discussed above, the motion-blur adjusting unit 106 is able to adjustthe amount of motion blur of the foreground image containing motion blurbased on the motion vector and the area information.

That is, it is possible to adjust the amount of motion blur contained inthe pixel values, that is, contained in sampled data.

As is seen from the foregoing description, the signal processingapparatus shown in FIG. 2 is able to adjust the amount of motion blurcontained in the input image. The signal processing apparatus configuredas shown in FIG. 2 is able to calculate the mixture ratio α, which isembedded information, and outputs the calculated mixture ratio α.

FIG. 79 is a block diagram illustrating another example of theconfiguration of the motion-blur adjusting unit 106. The motion vectorand the positional information thereof supplied from the motion detector102 are supplied to a unit-of-processing determining portion 901 and anadjusting portion 905. The area information supplied from the areaspecifying unit 103 is supplied to the unit-of-processing determiningportion 901. The foreground component image supplied from theforeground/background separator 105 is supplied to a calculator 904.

The unit-of-processing determining portion 901 supplies, together withthe motion vector, the unit of processing generated based on the motionvector and the positional information thereof and the area informationto a model-forming portion 902.

The model-forming portion 902 forms a model based on the motion vectorand the input unit of processing. More specifically, for example, themodel-forming portion 902 may store in advance a plurality of models inaccordance with the number of pixels contained in the unit ofprocessing, the number of virtual divided portions of the pixel value inthe time direction, and the number of foreground components for eachpixel. The model-forming portion 902 then may select the model in whichthe correlation between the pixel values and the foreground componentsis designated, such as that in FIG. 80, based on the unit of processingand the number of virtual divided portions of the pixel value in thetime direction.

It is now assumed, for example, that the number of pixels correspondingto the unit of processing is 12, and that the amount of movement v is 5.Then, the model-forming portion 902 sets the number of virtual dividedportions to 5, and selects a model formed of eight types of foregroundcomponents so that the leftmost pixel contains one foreground component,the second pixel from the left contains two foreground components, thethird pixel from the left contains three foreground components, thefourth pixel from the left contains four pixel components, the fifthpixel from the left contains five foreground components, the sixth pixelfrom the left contains five foreground components, the seventh pixelfrom the left contains five foreground components, the eighth pixel fromthe left contains five foreground components, the ninth pixel from theleft contains four foreground components, the tenth pixel from the leftcontains three foreground components, the eleventh pixel from the leftcontains two foreground components, and the twelfth pixel from the leftcontains one foreground component.

Instead of selecting a model from the prestored models, themodel-forming portion 902 may generate a model based on the motionvector and the unit of processing when the motion vector and the unit ofprocessing are supplied.

An equation generator 903 generates an equation based on the modelsupplied from the model-forming portion 902.

A description is now given, with reference to the models of foregroundcomponent images shown in FIGS. 80 through 82, of an example of theequation generated by the equation generator 903 when the number offoreground components is 8, the number of pixels corresponding to theunit of processing is 12, and the amount of movement v is 5.

When the foreground components contained in the foreground componentimage corresponding to the shutter time/v are F01/v through F08/v, therelationships between F01/v through F08/v and pixel values C01 throughC12 can be expressed by equations (74) through (85), as stated above.

By considering the pixel values C12 and C11, the pixel value C12contains only the foreground component F08/v, as expressed by equation(107), and the pixel value C11 consists of the product sum of theforeground component F08/v and the foreground component F07/v.Accordingly, the foreground component F07/v can be found by equation(108).F08/v=C12  (107)F07/v=C11−C12  (108)

Similarly, by considering the foreground components contained in thepixel values C10 through C01, the foreground components F06/v throughF01/v can be found by equations (109) through (114), respectively.F06/v=C10−C11  (109)F05/v=C09−C10  (110)F04/v=C08−C09  (111)F03/v=C07−C08+C12  (112)F02/v=C06−C07+C11−C12  (113)F01/v=C05−C06+C10−C11  (114)

The equation generator 903 generates the equations for calculating theforeground components by the difference between the pixel values, asindicated by the examples of equations (107) through (114). The equationgenerator 903 supplies the generated equations to the calculator 904.

The calculator 904 sets the pixel values of the foreground componentimage in the equations supplied from the equation generator 903 so as toobtain the foreground components based on the equations in which thepixel values are set. For example, when equations (107) through (114)are supplied from the equation generator 903, the calculator 904 setsthe pixel values C05 through C12 in equations (107) through (114).

The calculator 904 calculates the foreground components based on theequations in which the pixel values are set. For example, the calculator904 calculates the foreground components F01/v through F08/v, as shownin FIG. 81, based on the calculations of equations (107) through (114)in which the pixel values C05 through C12 are set. The calculator 904supplies the foreground components F01/v through F08/v to the adjustingportion 905.

The adjusting portion 905 multiplies the foreground components suppliedfrom the calculator 904 by the amount of movement v contained in themotion vector supplied from the unit-of-processing determining portion901 so as to obtain the foreground pixel values from which motion bluris eliminated. For example, when the foreground components F01/v throughF08/v are supplied from the calculator 904, the adjusting portion 905multiples each of the foreground components F01/v through F08/v by theamount of movement v, i.e., 5, so as to obtain the foreground pixelvalues F01 through F08 from which motion blur is eliminated, as shown inFIG. 82.

The adjusting portion 905 supplies the foreground component imageconsisting of the foreground pixel values without motion blur calculatedas described above to a motion-blur adder 906 and a selector 907.

The motion-blur adder 906 is able to adjust the amount of motion blur byusing the amount v′ by which motion blur is adjusted, which is differentfrom the amount of movement v, for example, the amount v′ by whichmotion blur is adjusted, which is one half the value of the amount ofmovement v, or the amount v′ by which motion blur is adjusted, which isirrelevant to the amount of movement v. For example, as shown in FIG.75, the motion-blur adder 906 divides the foreground pixel value Fiwithout motion blur by the amount v′ by which motion blur is adjusted soas to obtain the foreground component Fi/v′. The motion-blur adder 906then calculates the sum of the foreground components Fi/v′, therebygenerating the pixel value in which the amount of motion blur isadjusted. For example, when the amount v′ by which motion blur isadjusted is 3, the pixel value C02 is set to (F01)/v′, the pixel valueC3 is set to (F01+F02)/v′, the pixel value C04 is set to(F01+F02+F03)/v′, and the pixel value C05 is set to (F02+F03+F04)/v′.

The motion-blur adder 906 supplies the foreground component image inwhich the amount of motion blur is adjusted to the selector 907.

The selector 907 selects, based on a selection signal reflecting auser's selection, either the foreground component image supplied fromthe adjusting portion 905 from which motion blur is eliminated or theforeground component image supplied from the motion-blur adder 906 inwhich the amount of motion blur is adjusted, and outputs the selectedforeground component image.

As discussed above, the motion-blur adjusting unit 106 is able to adjustthe amount of motion blur based on the selection signal and the amountv′ by which motion blur is adjusted.

The processing for adjusting the amount of motion blur of the foregroundexecuted by the motion-blur adjusting unit 106 configured as shown inFIG. 79 is described below with reference to the flowchart of FIG. 83.

In step S901, the unit-of-processing determining portion 901 of themotion-blur adjusting unit 106 generates the unit of processing based onthe motion vector and the area information, and supplies the generatedunit of processing to the model-forming portion 902 and the adjustingportion 905.

In step S902, the model-forming portion 902 of the motion-blur adjustingunit 106 selects or generates the model according to the amount ofmovement v and the unit of processing. In step S903, the equationgenerator 903 generates, based on the selected or generated model, theequations for calculating the foreground components by the differencebetween the pixel values of the foreground component image.

In step S904, the calculator 904 sets the pixel values of the foregroundcomponent image in the generated equations, and extracts the foregroundcomponents by using the difference between the pixel values based on theequations in which the pixel values are set. In step S905, thecalculator 904 determines whether all the foreground componentscorresponding to the unit of processing have been extracted. If it isdetermined that all the foreground components corresponding to the unitof processing have not been extracted, the process returns to step S904,and the processing for extracting the foreground components is repeated.

If it is determined in step S905 that all the foreground componentscorresponding to the unit of processing have been extracted, the processproceeds to step S906. In step S906, the adjusting portion 905 adjustseach of the foreground components F01/v through F08/v supplied from thecalculator 904 based on the amount of movement v so as to obtain theforeground pixel values F01/v through F08/v from which motion blur iseliminated.

In step S907, the motion-blur adder 906 calculates the foreground pixelvalues in which the amount of motion blur is adjusted, and the selector907 selects the image without motion blur or the image in which theamount of motion blur is adjusted, and outputs the selected image. Thenthe process ends.

As described above, the motion-blur adjusting unit 106 configured asshown in FIG. 79 is able to more speedily adjust motion blur of theforeground image containing motion blur according to simplercomputations.

A known technique for partially eliminating motion blur, such as aWiener filter, is effective when being used in the ideal state, but isnot sufficient for an actual image quantized and containing noise. Incontrast, it is proved that the motion-blur adjusting unit 106configured as shown in FIG. 79 is sufficiently effective for an actualimage quantized and containing noise. It is thus possible to eliminatemotion blur with high precision.

FIG. 84 is a block diagram illustrating another configuration of thefunction of the signal processing apparatus.

The elements similar to those shown in FIG. 2 are designated with thesame reference numerals, and an explanation thereof is thus omitted.

The area specifying unit 103 supplies area information to themixture-ratio calculator 104 and a synthesizer 1001.

The mixture-ratio calculator 104 supplies the mixture ratio α to theforeground/background separator 105 and the synthesizer 1001.

The foreground/background separator 105 supplies the foregroundcomponent image to the synthesizer 1001.

The synthesizer 1001 combines a certain background image with theforeground component image supplied from the foreground/backgroundseparator 105 based on the mixture ratio α supplied from themixture-ratio calculator 104 and the area information supplied from thearea specifying unit 103, and outputs the synthesized image in which thecertain background image and the foreground component image arecombined.

FIG. 85 illustrates the configuration of the synthesizer 1001. Abackground component generator 1021 generates a background componentimage based on the mixture ratio α and a certain background image, andsupplies the background component image to a mixed-area-imagesynthesizing portion 1022.

The mixed-area-image synthesizing portion 1022 combines the backgroundcomponent image supplied from the background component generator 1021with the foreground component image so as to generate a mixed-areasynthesized image, and supplies the generated mixture-area synthesizedimage to an image synthesizing portion 1023.

The image synthesizer 1023 combines the foreground component image, themixed-area synthesized image supplied from the mixed-area-imagesynthesizing portion 1022, and the certain background image based on thearea information so as to generate a synthesized image, and outputs it.

As discussed above, the synthesizer 1001 is able to combine theforeground component image with a certain background image.

The image obtained by combining a foreground component image with acertain background image based on the mixture ratio α, which is thefeature quantity, appears more natural compared to an image obtained bysimply combining pixels.

FIG. 86 is a block diagram illustrating another configuration of thefunction of the signal processing apparatus for adjusting the amount ofmotion blur. The signal processing apparatus shown in FIG. 2sequentially performs the area-specifying operation and the calculationfor the mixture ratio α. In contrast, the signal processing apparatusshown in FIG. 86 simultaneously performs the area-specifying operationand the calculation for the mixture ratio α.

The functional elements similar to those in the block diagram of FIG. 2are designated with the same reference numerals, and an explanationthereof is thus omitted.

An input image is supplied to a mixture-ratio calculator 1101, aforeground/background separator 1102, the area specifying unit 103, andthe object extracting unit 101.

The mixture-ratio calculator 1101 calculates, based on the input image,the estimated mixture ratio when it is assumed that each pixel containedin the input image belongs to the covered background area, and theestimated mixture ratio when it is assumed that each pixel contained inthe input image belongs to the uncovered background area, and suppliesthe estimated mixture ratios calculated as described above to theforeground/background separator 1102.

FIG. 87 is a block diagram illustrating an example of the configurationof the mixture-ratio calculator 1101.

An estimated-mixture-ratio processor 401 shown in FIG. 87 is the same asthe estimated-mixture-ratio processor 401 shown in FIG. 46. Anestimated-mixture-ratio processor 402 shown in FIG. 87 is the same asthe estimated-mixture-ratio processor 402 shown in FIG. 46.

The estimated-mixture-ratio processor 401 calculates the estimatedmixture ratio for each pixel by the computation corresponding to a modelof the covered background area based on the input image, and outputs thecalculated estimated mixture ratio.

The estimated-mixture-ratio processor 402 calculates the estimatedmixture ratio for each pixel by the computation corresponding to a modelof the uncovered background area based on the input image, and outputsthe calculated estimated mixture ratio.

The foreground/background separator 1102 generates the foregroundcomponent image from the input image based on the estimated mixtureratio calculated when it is assumed that the pixel belongs to thecovered background area supplied from the mixture-ratio calculator 1101,the estimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and the area information supplied from the areaspecifying unit 103, and supplies the generated foreground componentimage to the motion-blur adjusting unit 106 and the selector 107.

FIG. 88 is a block diagram illustrating an example of the configurationof the foreground/background separator 1102.

The elements similar to those of the foreground/background separator 105shown in FIG. 63 are indicated by the same reference numerals, and anexplanation thereof is thus omitted.

A selector 1121 selects, based on the area information supplied from thearea specifying unit 103, either the estimated mixture ratio calculatedwhen it is assumed that the pixel belongs to the covered background areasupplied from the mixture-ratio calculator 1101 or the estimated mixtureratio calculated when it is assumed that the pixel belongs to theuncovered background area supplied from the mixture-ratio calculator1101, and supplies the selected estimated mixture ratio to theseparating portion 601 as the mixture ratio α.

The separating portion 601 extracts the foreground components and thebackground components from the pixel values of the pixels belonging tothe mixed area based on the mixture ratio α supplied from the selector1121 and the area information, and supplies the extracted foregroundcomponents to the synthesizer 603 and also supplies the backgroundcomponents to the synthesizer 605.

The separating portion 601 can be configured similarly to thecounterpart shown in FIG. 68.

The synthesizer 603 synthesizes the foreground component image andoutputs it. The synthesizer 605 synthesizes the background componentimage and outputs it.

The motion-blur adjusting unit 106 shown in FIG. 86 can be configuredsimilarly to the counterpart shown in FIG. 2. The motion-blur adjustingunit 106 adjusts the amount of motion blur contained in the foregroundcomponent image supplied from the foreground/background separator 1102based on the area information and the motion vector, and outputs theforeground component image in which the amount of motion blur isadjusted.

The selector 107 shown in FIG. 86 selects, based on, for example, aselection signal reflecting a user's selection, the foreground componentimage supplied from the foreground/background separator 1102 or theforeground component image supplied from the motion-blur adjusting unit106 in which the amount of motion blur is adjusted, and outputs theselected foreground component image.

As discussed above, the signal processing apparatus shown in FIG. 86 isable to adjust the amount of motion blur contained in an imagecorresponding to a foreground object of the input image, and outputs theresulting foreground object image. As in the first embodiment, thesignal processing apparatus configured as shown in FIG. 86 is able tocalculate the mixture ratio α, which is embedded information, andoutputs the calculated mixture ratio α.

FIG. 89 is a block diagram illustrating another configuration of thefunction of the signal processing apparatus for combining a foregroundcomponent image with a certain background image. The signal processingapparatus shown in FIG. 84 performs the area-specifying operation andthe calculation for the mixture ratio α in a serial manner. In contrast,the signal processing apparatus shown in FIG. 89 performs thearea-specifying operation and the calculation for the mixture ratio α ina parallel manner.

The functional elements similar to those indicated by the block of FIG.86 are indicated by the same reference numerals, and an explanationthereof is thus omitted.

The mixture-ratio calculator 1101 shown in FIG. 89 calculates, based onthe input image, the estimated mixture ratio when it is assumed thateach pixel contained in the input image belongs to the coveredbackground area, and the estimated mixture ratio when it is assumed thateach pixel contained in the input image belongs to the uncoveredbackground area, and supplies the estimated mixture ratios calculated asdescribed above to the foreground/background separator 1102 and asynthesizer 1201.

The foreground/background separator 1102 shown in FIG. 89 generates theforeground component image from the input image based on the estimatedmixture ratio calculated when it is assumed that the pixel belongs tothe covered background area supplied from the mixture-ratio calculator1101, the estimated mixture ratio calculated when it is assumed that thepixel belongs to the uncovered background area supplied from themixture-ratio calculator 1101, and the area information supplied fromthe area specifying unit 103, and supplies the generated foregroundcomponent image to the synthesizer 1201.

The synthesizer 1201 combines a certain background image with theforeground component image supplied from the foreground/backgroundseparator 1102 based on the estimated mixture ratio calculated when itis assumed that the pixel belongs to the covered background areasupplied from the mixture-ratio calculator 1101, the estimated mixtureratio calculated when it is assumed that the pixel belongs to theuncovered background area supplied from the mixture-ratio calculator1101, and the area information supplied from the area specifying unit103, and outputs the synthesized image in which the certain backgroundimage and the foreground component image are combined.

FIG. 90 illustrates the configuration of the synthesizer 1201. Thefunctional elements similar to those of the block diagram of FIG. 85 aredesignated with the same reference numerals, and explanation thereof isthus omitted.

A selector 1221 selects, based on the area information supplied from thearea specifying unit 103, either the estimated mixture ratio calculatedwhen it is assumed that the pixel belongs to the covered background areasupplied from the mixture-ratio calculator 1101 or the estimated mixtureratio calculated when it is assumed that the pixel belongs to theuncovered background area supplied from the mixture-ratio calculator1101, and supplies the selected estimated mixture ratio to thebackground component generator 1021 as the mixture ratio α.

The background component generator 1021 shown in FIG. 90 generates abackground component image based on the mixture ratio α supplied fromthe selector 1221 and a certain background image, and supplies thebackground component image to the mixed-area-image synthesizing portion1022.

The mixed-area-image synthesizing portion 1022 shown in FIG. 90 combinesthe background component image supplied from the background componentgenerator 1021 with the foreground component image so as to generate amixed-area synthesized image, and supplies the generated mixed-areasynthesized image to the image synthesizing portion 1023.

The image synthesizing portion 1023 combines the foreground componentimage, the mixed-area synthesized image supplied from themixed-area-image synthesizing portion 1022, and the background imagebased on the area information so as to generate a synthesized image andoutputs it.

In this manner, the synthesizer 1201 is able to combine the foregroundcomponent image with a certain background image.

The embodiment has been discussed above by setting the mixture ratio αto the ratio of the background components contained in the pixel values.However, the mixture ratio α may be set to the ratio of the foregroundcomponents contained in the pixel values.

The embodiment has been discussed above by setting the moving directionof the foreground object to the direction from the left to the right.However, the moving direction is not restricted to the above-describeddirection.

In the above description, a real-space image having a three-dimensionalspace and time axis information is projected onto a time space having atwo-dimensional space and time axis information by using a video camera.However, the present invention is not restricted to this example, andcan be applied to the following case. When a greater amount of firstinformation in one-dimensional space is projected onto a smaller amountof second information in a two-dimensional space, distortion generatedby the projection can be corrected, significant information can beextracted, or a more natural image can be synthesized.

The sensor used herein is not restricted to a CCD, and may be anothertype of sensor, such as a solid-state image-capturing device, forexample, a BBD (Bucket Brigade Device), a CID (Charge Injection Device),or a CPD (Charge Priming Device), or a CMOS (Complementary Metal OxideSemiconductor). Also, the sensor does not have to be a sensor in whichdetection devices are arranged in a matrix, and may be a sensor in whichdetection devices are arranged in one line.

A recording medium in which a program for performing the signalprocessing of the present invention is recorded may be formed of apackage medium in which the program is recorded, which is distributedfor providing the program to a user separately from the computer, asshown in FIG. 1, such as the magnetic disk 51 (including a floppy(registered trade name) disk), the optical disc 52 (CD-ROM (CompactDisc-Read Only Memory) and a DVD (Digital Versatile Disc)), themagneto-optical disk 53 (including MD (Mini-Disc) (registered tradename)), or the semiconductor memory 54. The recording medium may also beformed of the ROM 22 or a hard disk contained in the storage unit 28 inwhich the program is recorded, such recording medium being provided tothe user while being prestored in the computer.

The steps forming the program recorded in a recording medium may beexecuted chronologically according to the orders described in thespecification. However, they do not have to be executed in a time-seriesmanner, and they may be executed concurrently or individually.

INDUSTRIAL APPLICABILITY

According to the present invention, a region in which mixture occurs canbe detected.

1. An image processing apparatus to detect a mixed area from image datawhich is formed of a predetermined number of pieces of pixel dataobtained by an image-capturing device including a predetermined numberof pixels, the pixels having a time integrating function, the mixed areabeing obtained as the pixel data in which a plurality of objects aremixed in the real world, said image processing apparatus comprising: amotion compensation unit configured to compensate for the motion offrames of the image data; and an area detector configured to subtractpixel data at a corresponding position in the motion-compensated frames,determine a difference between the subtracted pixel data, and detect themixed area based on the difference between the pixel data at thecorresponding position in the motion-compensated frames, wherein saidarea detector detects the mixed area to which the pixel data belongswhen the difference is greater than or equal to a threshold, wherein ashutter-time period is divided into a plurality of equal periods, eachpixel data being divided in accordance with the plurality of periods andpixel data corresponding to each of the plurality of equal periods beingobtained for estimating a mixed ratio of each pixel in the mixed area,and wherein the motion compensation unit generates themotion-compensated frames by matching backgrounds of the image data. 2.The image processing apparatus according to claim 1, wherein said areadetector further detects, based on temporal change of the detected mixedarea, a covered background area in which a foreground object componentof the objects corresponding to a foreground increases over time and anuncovered background area in which a background object component of theobjects corresponding to a background increases over time.
 3. The imageprocessing apparatus according to claim 1, wherein said area detectorfurther detects, based on a motion vector corresponding to the pixeldata in each of the frames, a covered background area in which aforeground object component of the objects corresponding to a foregroundincreases over time and an uncovered background area in which abackground object component of the objects corresponding to a backgroundincreases over time.
 4. The image processing apparatus according toclaim 3, further comprising a motion vector detector configured todetect the motion vector.
 5. The image processing apparatus according toclaim 1, further comprising a mixture-ratio calculation unit configuredto detect a mixture ratio indicating the state in which the objects aremixed in the pixel data.
 6. The image processing apparatus according toclaim 5, further comprising a separation unit configured to separate atleast a foreground object component of the objects corresponding to aforeground from the pixel data of the mixed area based on the mixtureratio.
 7. The image processing apparatus according to claim 6, furthercomprising a motion-blur adjusting unit configured to adjust the amountof motion blur in the separated foreground object component.
 8. Theimage processing apparatus according to claim 6, further comprising asynthesizing unit configured to synthesize a desired object with theseparated foreground object component based on the mixture ratio.
 9. Theimage processing apparatus according to claim 1, wherein said motioncompensation unit performs motion compensation by shifting a peripheralframe around a designated frame so that a background object of theplurality of objects in the designated frame is disposed at the samepixel position as the background object in the peripheral frame; andsaid area detector detects at least the mixed area based on thedifference between the motion-compensated peripheral frame and thedesignated frame.
 10. The image processing apparatus according to claim9, wherein said area detector includes a stationary/moving determinationunit configured to perform a stationary or moving determination based onthe difference between the pixel data at the corresponding pixelposition in the motion-compensated peripheral frame and the designatedframe; and said area detector detects, based on the determination ofsaid stationary/moving determination unit, in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.
 11. The image processing apparatus accordingto claim 10, wherein said area detector specifies an uncoveredbackground area and a covered background area in the mixed area based onthe determination of said stationary/moving determination unit, theuncovered background area being formed at the trailing end in thedirection in which the foreground object is moving, the coveredbackground area being formed at the leading end in the direction inwhich the foreground object is moving.
 12. An image processing methodfor detecting a mixed area from image data which is formed of apredetermined number of pieces of pixel data obtained by animage-capturing device including a predetermined number of pixels, thepixels having a time integrating function, the mixed area being obtainedas the pixel data in which a plurality of objects are mixed in the realworld, said image processing method comprising: a motion compensatingstep of compensating for the motion of frames of the image data; and anarea detecting step of subtracting pixel data at a correspondingposition in the motion-compensated frames, determining a differencebetween the subtracted pixel data, and detecting the mixed area based onthe difference between the pixel data at the corresponding position inthe motion-compensated frames, wherein, in said area detecting step, themixed area to which at least the pixel data belongs is detected when thedifference is greater than or equal to a threshold, wherein ashutter-time period is divided into a plurality of equal periods, eachpixel data being divided in accordance with the plurality of periods andpixel data corresponding to each of the plurality of equal periods beingobtained for estimating a mixed ratio of each pixel in the mixed area,and wherein the motion compensating step generates themotion-compensated frames by matching backgrounds of the image data. 13.The image processing method according to claim 12, wherein, in said areadetecting step, a covered background area in which a foreground objectcomponent of the objects corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent of the objects corresponding to a background increases overtime are further detected based on temporal change of the detected mixedarea.
 14. The image processing method according to claim 12, wherein, insaid area detecting step, a covered background area in which aforeground object component of the objects corresponding to a foregroundincreases over time and an uncovered background area in which abackground object component of the objects corresponding to a backgroundincreases over time are further detected based on a motion vectorcorresponding to the pixel data in each of the frames.
 15. The imageprocessing method according to claim 14, further comprising a motionvector detecting step of detecting the motion vector.
 16. The imageprocessing method according to claim 12, further comprising amixture-ratio calculating step of calculating a mixture ratio indicatingthe state in which the objects are mixed in the pixel data.
 17. Theimage processing method according to claim 16, further comprising aseparating step of separating at least a foreground object component ofthe objects corresponding to a foreground from the pixel data of themixed area based on the mixture ratio.
 18. The image processing methodaccording to claim 17, further comprising a motion-blur adjusting stepof adjusting the amount of motion blur in the separated foregroundobject component.
 19. The image processing method according to claim 17,further comprising a synthesizing step of synthesizing another desiredobject with the separated foreground object component based on themixture ratio.
 20. The image processing method according to claim 12,wherein, in said motion compensating step, motion compensation isperformed by shifting a peripheral frame around a designated frame sothat a background object of the plurality of objects in the designatedframe is disposed at the same pixel position as the background object inthe peripheral frame; and in said area detecting step, at least themixed area is detected based on the difference between themotion-compensated peripheral frame and the designated frame.
 21. Theimage processing method according to claim 20, wherein said areadetecting step includes a stationary/moving determining step ofperforming a stationary or moving determination based on the differencebetween the pixel data at the corresponding pixel position in themotion-compensated peripheral frame and the designated frame; and insaid area detecting step, is detected based on a determination of aforeground area formed of only a foreground object component forming theforeground object in the plurality of objects, a background area formedof only a background object component forming the background object, orthe mixed area the pixel position is.
 22. The image processing methodaccording to claim 21, wherein, in said area detecting step, anuncovered background area and a covered background area in the mixedarea are specified, the uncovered background area being formed at thetrailing end in the direction in which the foreground object is moving,the covered background area being formed at the leading end in thedirection in which the foreground object is moving.
 23. A computerreadable recording medium encoded with a computer-readable program forimage processing, the program being adapted to detect a mixed area fromimage data which is formed of a predetermined number of pieces of pixeldata obtained by an image-capturing device including a predeterminednumber of pixels, the pixels having a time integrating function, themixed area being obtained as the pixel data in which a plurality ofobjects are mixed in the real world, the program comprising: a motioncompensating step of compensating for the motion of frames of the imagedata; and an area detecting step of subtracting pixel data at acorresponding position in the motion-compensated frames, determining adifference between the subtracted pixel data, and detecting the mixedarea based on the difference between the pixel data at the correspondingposition in the motion-compensated frames, wherein, in said areadetecting step, the mixed area to which at least the pixel data belongsis detected when the difference is greater than or equal to a threshold,wherein a shutter-time period is divided into a plurality of equalperiods, each pixel data being divided in accordance with the pluralityof periods and pixel data corresponding to each of the plurality ofequal periods being obtained for estimating a mixed ratio of each pixelin the mixed area, and wherein the motion compensating step generatesthe motion-compensated frames by matching backgrounds of the image data.24. The recording medium according to claim 23, wherein, in said areadetecting step, a covered background area in which a foreground objectcomponent of the objects corresponding to a foreground increases overtime and an uncovered background area in which a background objectcomponent of the objects corresponding to a background increases overtime are further detected based on temporal change of the detected mixedarea.
 25. The recording medium according to claim 23, wherein, in saidarea detecting step, a covered background area in which a foregroundobject component of the objects corresponding to a foreground increasesover time and an uncovered background area in which a background objectcomponent of the objects corresponding to a background increases overtime are further detected based on a motion vector corresponding to thepixel data in each of the frames.
 26. The recording medium according toclaim 25, wherein the program further comprises a motion vectordetecting step of detecting the motion vector.
 27. The recording mediumaccording to claim 23, wherein the program further comprises amixture-ratio calculating step of calculating a mixture ratio indicatingthe state in which the objects are mixed in the pixel data.
 28. Therecording medium according to claim 27, wherein the program furthercomprises a separating step of separating at least a foreground objectcomponent of the objects corresponding to a foreground from the pixeldata of the mixed area based on the mixture ratio.
 29. The recordingmedium according to claim 28, wherein the program further comprises amotion-blur adjusting step of adjusting the amount of motion blur in theseparated foreground object component.
 30. The recording mediumaccording to claim 28, wherein the program further comprises asynthesizing step of synthesizing another desired object with theseparated foreground object component based on the mixture ratio. 31.The recording medium according to claim 23, wherein, in said motioncompensating step, motion compensation is performed by shifting aperipheral frame around a designated frame so that a background objectin the designated frame is disposed at the same pixel position as thebackground object in the peripheral frame; and in said area detectingstep, at least the mixed area is detected based on the differencebetween the motion-compensated peripheral frame and the designatedframe.
 32. The recording medium according to claim 31, wherein said areadetecting step includes a stationary/moving determining step ofperforming a stationary or moving determination based on the differencebetween the pixel data at the corresponding pixel position in themotion-compensated peripheral frame and the designated frame; and insaid area detecting step, it is detected based on the determination ofsaid stationary/moving determination means in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.
 33. The recording medium according to claim32, wherein, in said area detecting step, an uncovered background areaand a covered background area in the mixed area are specified based onthe determination of said stationary/moving determination means, theuncovered background area being formed at the trailing end in thedirection in which the foreground object is moving, the coveredbackground area being formed at the leading end in the direction inwhich the foreground object is moving.
 34. An image-capturing apparatuscomprising: an image-capturing unit configured to output a subject imagecaptured by an image-capturing device including a predetermined numberof pixels as image data consisting of a predetermined number of piecesof pixel data, the pixels having a time integrating function; a motioncompensation unit configured to compensate for the motion of frames ofthe image data; and an area detector configured to subtract pixel dataat a corresponding position in the motion-compensated frames, determinea difference between the subtracted pixel data, and detect from theimage data a mixed area as the pixel data in which a plurality ofobjects are mixed in the real world based on the difference between thepixel data at the corresponding position in the motion-compensatedframes, wherein said area detector detects the mixed area to which atleast the pixel data belongs when the difference is greater than orequal to a threshold, wherein a shutter-time period is divided into aplurality of equal periods, each pixel data being divided in accordancewith the plurality of periods and pixel data corresponding to each ofthe plurality of equal periods being obtained for estimating a mixedratio of each pixel in the mixed area, and wherein the motioncompensation unit generates the motion-compensated frames by matchingbackgrounds of the image data.
 35. The image-capturing apparatusaccording to claim 34, wherein said area detector further detects, basedon temporal change of the detected mixed area, a covered background areain which a foreground object component of the objects corresponding to aforeground increases over time and an uncovered background area in whicha background object component of the objects corresponding to abackground increases over time.
 36. The image-capturing apparatusaccording to claim 34, wherein said area detector further detects, basedon a motion vector corresponding to the pixel data in each of theframes, a covered background area in which a foreground object componentof the objects corresponding to a foreground increases over time and anuncovered background area in which a background object component of theobjects corresponding to a background increases over time.
 37. Theimage-capturing apparatus according to claim 36, further comprising amotion vector detector configured to detect the motion vector.
 38. Theimage-capturing apparatus according to claim 34, further comprisingmixture-ratio calculation unit configured to calculate a mixture ratioindicating the state in which the objects are mixed in the pixel data.39. The image-capturing apparatus according to claim 38, furthercomprising separation unit configured to separate at least a foregroundobject component of the objects corresponding to a foreground from thepixel data of the mixed area based on the mixture ratio.
 40. Theimage-capturing apparatus according to claim 39, further comprisingmotion-blur adjusting unit configured to adjust the amount of motionblur in the separated foreground object component.
 41. Theimage-capturing apparatus according to claim 39, further comprisingsynthesizing unit configured to synthesize another desired object withthe separated foreground object component based on the mixture ratio.42. The image-capturing apparatus according to claim 34, wherein saidmotion compensation unit performs motion compensation by shifting aperipheral frame around a designated frame so that a background objectof the plurality of objects in the designated frame is disposed at thesame pixel position as the background object in the peripheral frame;and said area detector detects at least the mixed area based on thedifference between the motion-compensated peripheral frame and thedesignated frame.
 43. The image-capturing apparatus according to claim42, wherein said area detector includes stationary/moving determinationunit configured to perform a stationary or moving determination based onthe difference between the pixel data at the corresponding pixelposition in the motion-compensated peripheral frame and the designatedframe; and said area detector detects, based on the determination ofsaid stationary/moving determination unit, in which of a foreground areaformed of only a foreground object component forming the foregroundobject in the plurality of objects, a background area formed of only abackground object component forming the background object, or the mixedarea the pixel position is.
 44. The image-capturing apparatus accordingto claim 43, wherein said area detector specifies an uncoveredbackground area and a covered background area in the mixed area based onthe determination of said stationary/moving determination unit, theuncovered background area being formed at the trailing end in thedirection in which the foreground object is moving, the coveredbackground area being formed at the leading end in the direction inwhich the foreground object is moving.